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FOREWORD 


Classification  is  the  process  of  assigning  new  enlisted  personnel  to  initial  job  training  in 
the  Army.  Investigations  of  improved  methods  for  doing  this  have  been  a  prominent  part  of  the 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  research  program 
since  shortly  after  World  War  II.  The  immediate  antecedent  of  this  work  was  the  ARI  Project  B 
research,  conducted  over  the  1982  -  1989  period,  which  led  to  the  testing  of  a  mainframe 
prototype.  PC  prototype  development  began  in  the  fall  of  1993  and  was  largely  completed  by 
the  spring  of  1997,  at  which  time  the  Deputy  Chief  of  Staff  for  Personnel  (DCSPER) 
recommended  that  ARI  continue  the  work  and  move  toward  implementation.  Army 
management  reviewed  the  Functional  Description  (FD)  in  the  Fall  of  1998,  and  the  Director  of 
Military  Personnel  Management  (DMPM)  recommended  that  ARI  conduct  a  field  test 
evaluation.  The  evaluation  is  scheduled  for  the  2001  -  2003  period. 

The  Army  currently  takes  a  minimum  enlistment  standards  approach  to  classification. 
EPAS,  working  as  a  subsystem  of  REQUEST  (the  Army’s  training  reservation  system),  is  an 
attempt  to  go  beyond  minimum  standards  and  make  better  use  of  each  recruit’s  potential. 
Laboratory  simulation  testing  of  the  prototype  models  indicated  the  likelihood  of  sizeable  gains 
in  classification  efficiency,  and  the  objective  of  the  field  test  is  to  confirm  these  gains  in  the 
presence  of  real-world  constraints  and  decision-making.  This  report  describes  the  design  for  the 
field  test  evaluation. 

The  focus  of  the  Selection  and  Assignment  Research  Unit  of  ARI  is  conducting  research, 
studies,  and  analysis  on  the  measurement  of  aptitudes  and  performance  of  individuals,  in  order  to 
improve  Army  selection,  classification,  retention,  and  promotion  of  officers  and  enlisted 
Soldiers.  This  study  will  provide  the  foundation  for  recommended  improved  aptitude 
measurement  and  classification  procedures  for  enlisted  personnel. 


SCOTT  E.  GRAHAM 
Acting  Technical  Director 
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Off-Line  Field  Test  Design  for  Evaluating  Two  Approaches  to  Person-Job  Matching:  The  Army 
Recruit  Quota  System  (REQUEST)  and  the  Enlisted  Personnel  Allocation  System  (EPAS) 

EXECUTIVE  SUMMARY 


Background: 

Classification  is  the  matching  of  recruits  into  their  entry  job  training.  The  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  has  been  conducting  research 
and  studies  into  better  classification  methods  and  developing  the  Enlisted  Personnel  Allocation 
System  (EPAS),  with  the  aim  of  enhancing  the  Army’s  current  training  reservation  system, 
known  as  REQUEST.  EPAS  is  designed  to  enhance  REQUEST  by  introducing  optimization 
methods  into  what  is  a  sequential  assignment  process.  This  is  done  by  treating  the  assignment 
process  as  two  phases.  In  the  first  phase,  a  linear  programming  model  represents  the  (forecasted) 
monthly  flow  of  applicants  and  availability  of  training  class  seats  over  the  recruiting  year. 
Applicants  are  categorized  into  supply  groups  by  their  demographics  and  aptitude  profiles.  The 
optimal  allocation  or  matching  of  (applicant)  supply  groups  to  military  occupational  specialty 
(MOS)  training  classes  is  determined.  The  optimal  allocation  is  the  one  that  maximizes 
predicted  performance  for  an  annual  accession  cohort,  while  meeting  accession  and  training 
management  goals.  The  model  solution  is  updated  weekly  and  used  to  generate  an  ordered  list  of 
MOS  training  recommendations  that  best  match  each  supply  group  with  training  requirements. 

In  the  second  phase,  that  of  actual  applicant  assignment,  these  recommendations  are  merged  with 
those  generated  by  existing  REQUEST  procedures  and  presented  to  the  applicant  by  the  career 
counselor. 

Research  Requirement: 

A  large-scale  ARI  research  effort  called  Project  B  explored  alternative  approaches  to  the 
Army  classification  issue,  and  led  to  the  development  in  late  1980’s  of  a  mainframe-based  EPAS 
prototype.  This  work  was  continued  in  the  mid  -  1990’s  with  the  development  and  laboratory 
testing  of  a  PC-based  EPAS  prototype.  Following  the  recommendations  of  Army  management 
in  1998,  ARI  is  conducting  a  field  test  evaluation  of  the  EPAS  subsystem  over  the  2001  -  2003 
period.  The  objective  of  the  “field  test”  is  to  examine  the  efficacy  of  EPAS  with  as  much 
operational  realism  as  possible  while  having  no  effect  whatsoever  upon  USAREC  business 
transactions. 

Findings: 

To  accomplish  the  objective,  ARI  has  designed  a  non-intrusive  field  test  using 
operational  transactions  data  (between  potential  recruit  and  career  counselor)  within  a  simulation 
framework.  The  REQUEST  system  provided  the  EPAS  project  with  transaction  data  —  all  the 
training  opportunities  presented  to  each  applicant  -  over  the  FY  2002  period.  In  the  “fixed  list” 
design  alternative,  the  EPAS  recommendations  are  merged  with  the  actual  training  opportunities, 
and  the  applicant’s  choice  from  the  merged  list  is  made  (simulated)  using  a  statistical  job  choice 
model.  Each  simulation  is  comprised  of  thirty  replications.  The  fixed  design  constrains  the 
potential  benefits  that  could  be  realized  by  EPAS,  and  so  a  “dynamic  list”  design  is  also 
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of  the  original  REQUEST  list;  this  facilitates  a  more  accurate  portrayal  of  the  cumulative  effects 
of  EPAS  over  the  fiscal  year. 

Results  of  the  simulations  will  be  employed  to  address  the  following  major  operational 

issues: 

(1)  Classification  efficiency  of  EP  AS-enhanced  REQUEST  (EER).  The  metric  of  merit 
is  the  mean  predicted  performance  (MPP)  of  recruits  based  on  their  (Aptitude  Area  Composite 
scores  corresponding  to  the)  MOS  assignments  under  REQUEST  and  EER  classification 
systems.  The  analysis  is  designed  to  compare  the  overall  classification  benefits  of  EER  and 
REQUEST,  and  to  examine  whether  classification  efficiency  is  disproportionately  biased 
towards  certain  MOS  to  the  disadvantage  of  other  MOS. 

(2)  Impact  of  EPAS  upon  the  training  opportunities  presented  to  applicants  (i.e.,  job  list 
analysis).  The  analysis  is  designed  to  address:  How  large  is  the  intersection  between  the 
REQUEST  and  the  EPAS  list?  How  is  the  size  of  the  intersection  affected  by  the  date  of 
training?  What  is  the  frequency  of  priority  MOS  at  the  “top”  of  the  EER  list? 

(3)  Impact  of  EPAS  upon  meeting  overall  and  priority  accession  requirements.  The 
analysis  is  designed  to  address:  How  does  EER  compare  to  REQUEST  in  meeting  overall  Army 
monthly  accession  goals?  How  does  EER  compare  to  REQUEST  in  meeting  accession  goals  for 
priority  MOS? 

Within  each  of  these  areas,  this  design  document  specifies  indexes  with  which  to  measure 
the  impacts.  The  key  role  and  specification  of  the  job  choice  model  is  also  fully  described. 

Utilization  of  Findings: 

As  its  name  implies,  the  field  test  evaluation  design  described  in  this  report  provides  the 
conceptual  and  theoretical  underpinnings  for  the  simulation  approach,  analysis,  and  evaluation  of 
the  field  test. 
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I.  INTRODUCTION 


The  Enlisted  Personnel  Allocation  System  (EPAS),  initially  developed  through  a  multi¬ 
year  research  and  development  project  conducted  by  the  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI),  is  the  latest  tool  available  to  the  Army  for  improving  the 
classification  process.1  Designed  to  be  a  subsystem  of  the  Recruit  Quota  System  (REQUEST), 
EPAS  is  a  person-job-matching  (PJM)  method  that  optimizes  the  assignment  of  recruits  to  entry- 
level  military  occupational  specialty  (MOS)  training.  It  goes  beyond  REQUEST,  the  Army’s 
present  approach  to  PJM  (see  Figure  1).  REQUEST  identifies  high  priority  MOS  for  which  an 
applicant  meets  the  minimum  Aptitude  Area  (AA)  composite  score  qualifications.  (Applicant 
composite  scores  are  derived  from  military  entrance  testing  known  as  the  Armed  Services 
Vocational  Aptitude  Battery  -  ASVAB.)  In  addition,  EPAS  identifies  those  MOS  in  which  an 
individual  is  likely  to  perform  with  the  greatest  effectiveness,  while  meeting  overall  Army 
accession  goals  and  filling  critical  MOS. 

A  PC-EPAS  prototype  was  created  and  evaluated  based  on  laboratory  simulations  of  the 
Army’s  classification  process  in  FY  1998.  The  results  of  laboratory  classification  using  the 
EPAS  Optimal  Guidance  (EOG)  list  provided  evidence  that  EPAS  can  improve  the  mean 
predicted  performance  (measured  as  the  average  AA  composite  score  of  recruits  in  their  assigned 
job  training)  of  a  fiscal  year  recruit  cohort,  while  simultaneously  meeting  Army  enlistment 
requirements.  Based  on  these  positive  laboratory  results,  ARI  developed  a  production  version  of 
EPAS  in  FY  2000.  The  planned  field  test  will  examine  the  likelihood  of  realizing  the  laboratory 
findings  in  an  “operational”  environment,  using  EPAS  linked  as  a  subsystem  to  REQUEST 
within  a  simulation  framework.  This  report  describes  the  planned  field  test  design. 

Background 


EPAS  Optimal  Guidance 

EPAS  produces  a  separate  list  of  MOS  training  assignments  for  each  of  127  recruit 
Supply  Groups,  which  are  groups  of  recruits  based  on  demographic  characteristics  and  ASVAB 
subtest  score  profiles.  The  list  that  EPAS  outputs  is  called  the  EOG  or  EPAS  Optimal  Guidance, 
and  is  also  referred  elsewhere  in  this  document  as  the  EPAS  list.  The  MOS  job  training 
opportunities  in  this  list  are  ranked  from  high  to  low  in  terms  of  an  EPAS  Index  of  System 
Efficiency.  This  index  reflects  the  estimated  Aptitude  Area  Composite  score  of  the  recruit 
Supply  Group  for  each  MOS,  and  the  forecast  of  contracts  and  MOS  training  requirements  over 
the  planning  horizon.  Since  EPAS  would  be  updated  daily  or  weekly,  the  guidance  weights 
current  Army  enlistment  needs  most  heavily.  However,  it  also  accounts  for  the  “big  picture,” 
which  includes  the  results  of  assignments  already  made  and  the  forecasts  of  contracts  and  MOS 
training  requirements  for  the  current  and  following  fiscal  year. 

EP AS-enhanced  REQUEST  List 


1  This  section  draws  on  M.A.  Lightfoot  and  P.  Ramsberger  (2000),  Matching  Recruits  to  Jobs:  The  Enlisted 
Personnel  Allocation  System  (Special  Report  41).  Alexandria,  VA:  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences. 
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The  present  notion  for  integrating  EPAS  as  a  subsystem  of  REQUEST  is  to  run  the  EPAS 
PJM  process  along  side  the  PJM  process  of  REQUEST  as  sh  -A  n  in  Figure  2.  The  EPAS  Optimal 
Guidance  assignment  list  for  a  selected  Recruit  Supply  Group  would  be  merged  with  the 
REQUEST  assignment  list  for  a  particular  recruit  who  fits  that  Supply  Group  profile.  The  EPAS 
Optimal  Guidance  would  be  used  to  reorder  the  REQUEST  assignments  from  highest  to  lowest 


Figure  1.  Graphical  Depiction  of  the  Operational  REQUEST  Classification  System 


Figure  2.  Graphical  Depiction  of  the  Proposed  Operational  EPAS-enhanced  REQUEST 
Classification  System 


in  terms  of  an  EPAS  Index  of  System  Efficiency.  This  approach  would  capitalize  upon  the  EPAS 
near-optimal  batch  processing  technique  based  on  Army  contract  missions  and  MOS  training 
requirements. 

What  will  help  ensure  the  greatest  possible  overlap  between  EPAS  Optimal  Guidance  and 
REQUEST  assignment  lists  is  that  the  EPAS  optimization  algorithm  also  takes  into  account 
Army-wide  enlistment  requirements,  such  as  meeting  critical  and  hard-to-fill  MOS  needs  and 
balancing  the  distribution  of  highly  qualified  recruits  across  all  MOS.  Where  there  are 
discrepancies  between  the  two  lists  of  assignments,  the  non-overlapping  EPAS  assignments 
would  be  dropped,  while  the  non-overlapping  REQUEST  assignments  would  be  retained  and 
placed  at  the  bottom  of  the  list  in  the  order  in  which  they  were  output  by  REQUEST.  This  will 
help  to  ensure  that  critical  enlistment  requirements  are  given  high  priority  in  the  integrated 
REQUEST-EPAS  classification  system.  The  final  integrated  REQUEST-EPAS  merged  list  is 
also  referred  to  in  the  remainder  of  this  document  as  the  EP AS-enhanced  REQUEST  list. 

The  integrated  REQUEST-EPAS  classification  procedure  has  potential  benefits  for  both 
the  Army  and  the  Soldier.  The  Army  will  gain  through  the  higher  potential  for  training  and  job 
success  of  all  recruits.  Soldiers  will  gain  because  better  matching  of  their  aptitudes  to  MOS 
requirements  creates  a  higher  potential  for  career  success.  When  a  recruit  cohort  is  taken  as  a 
whole,  the  hybrid  REQUEST-EPAS  assignment  process  is  expected  to  produce  high  quality 
assignments,  which  raise  the  average  Aptitude  Area  composite  score  across  all  MOS  and 
simultaneously  meet  critical  Army  enlistment  requirements. 

The  field  test  will  be  conducted  within  a  non-intrusive  simulation  framework  that  will 
maintain  a  high  degree  of  operational  realism.  The  integrated  REQUEST-EPAS  procedure, 
referred  to  elsewhere  in  this  document  as  the  EPAS-enhanced  REQUEST  (EER)  system,  is  the 
ultimate  focus  of  this  field  test.  Its  classification  efficiency  and  capability  to  meet  Army 
enlistment  requirements  will  be  compared  to  the  current  REQUEST  system  in  an  operational 
environment.  The  field  test  will  also  compare  the  MOS  training  opportunities  in  the  EPAS  list 
with  those  in  the  REQUEST  list,  in  order  to  examine  how  EPAS  impacts  recruit  training 
opportunities  in  an  operational  environment. 

The  organization  of  the  remainder  of  this  document  is  as  follows.  In  Chapter  2,  the  issues 
described  above  are  more  specifically  formulated  in  the  form  of  field  test  research  questions. 
Chapter  3  discusses  the  main  components  of  and  key  issues  in  the  proposed  off-line  field  test 
approach.  Chapter  4  provides  a  technical  description  of  the  indices  that  will  be  employed  in 
analyzing  the  research  questions,  and  presents  an  overview  of  the  approach  in  these  analyses. 
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II.  FIELD  TEST  STUDY  QUESTIONS 

In  this  chapter  we  present  the  underlying  study  questions  that  are  central  to  the  three  sets 
of  analyses  in  this  field  test.  We  discuss  how  each  of  these  three  analyses  is  important  to  the 
Army  and  briefly  describe  the  respective  indices  associated  with  the  questions  identified.  The 
detailed  technical  descriptions  of  the  indices  are  provided  in  Chapter  4. 

Job  List  Analysis 

The  job  list  analysis  is  important  to  the  Army  for  two  reasons.  First,  the  job  list 
formulated  for  a  recruit  by  the  REQUEST  and  EER  classification  systems  will  affect  the  range  of 
MOS  training  class  choices  that  the  Army  may  offer  this  recruit.  This  will  have  a  direct  effect  on 
a  recruit  finding  a  job  that  is  acceptable  and,  consequently,  his  or  her  likelihood  of  enlisting  in 
the  Army.  Second,  the  composition  of  the  EP AS-enhanced  MOS  list  will  have  practical 
implications  for  the  guidance  counselor’s  flexibility  in  selling  jobs  to  a  recruit.  This  is  important 
as  the  MOS  training  class  not  only  must  appeal  to  the  recruit,  but  must  be  consistent  with  Army 
priorities.  The  questions  proposed  below  guide  our  approach  to  evaluating  the  impact  of  the 
EPAS  enhancement  upon  MOS  training  choices  of  recruits. 

•  How  large  is  the  intersection  between  REQUEST  and  EPAS  list? 

The  intersection  will  be  measured  in  terms  of  the  count  or  proportion  of  common  MOS- 
class-month  choices  between  the  REQUEST  and  EPAS  lists  of  individual  recruits.  Two 
approaches  will  be  employed  to  construct  the  index  for  this  analysis.  The  first  approach  will  be 
based  on  the  simple  count  of  matching  MOS  training  opportunities  in  the  REQUEST  list  and  the 
EPAS  optimal  guidance  list  of  recruits.  The  second  approach  will  take  into  account  the 
respective  locations  of  matching  MOS  training  opportunities  in  the  REQUEST  list  and  EPAS  list 
of  recruits.  The  index  computed  in  the  second  approach  will  represent  the  “quality”  of  the 
intersection  between  the  two  lists.  Using  either  approach,  a  large  intersection  between 
REQUEST  and  EPAS  list  will  lead  to  greater  potential  EPAS  benefits.  As  an  extreme  example, 
if  the  intersection  between  the  two  lists  is  zero  then  EPAS  will  have  no  effect  on  the  MOS  job 
classifications  of  recruits,  as  the  EER  list  is  simply  the  original  REQUEST  list  itself. 

•  How  is  the  size  of  the  intersection  affected  by  the  date  of  training  class? 

In  this  question,  we  are  interested  in  finding  out  whether  the  MOS  contained  in 
REQUEST  and  EPAS  lists  match  but  class  start  dates  do  not.  The  indices  that  will  be  employed 
to  analyze  this  question  will  be  based  on  the  count  and  proportion  of  choices  in  the  REQUEST 
list  whose  MOS  appear  in  the  EPAS  list  for  the  recruit.  REQUEST  class  dates  and  EPAS  class 
months  will  be  ignored  when  matching  the  MOS  in  the  respective  lists.  In  this  analysis, 
intersections  that  are  substantially  larger  in  comparison  to  those  obtained  in  the  analysis  of  the 
preceding  research  question  would  indicate  that  EPAS  is  missing  potential  optimization  due  to 
mismatching  training  dates.  Results  from  this  analysis  will  have  implications  for  relating  the 
EPAS  model  optimization  to  REQUEST  DEP  controls. 
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•  What  is  the  frequency  of  priority  MOS  at  the  “top”  of  the  EP AS-enhanced  REQUEST  list? 

The  analysis  for  this  study  question  will  examine  to  what  extent  potential  EPAS 
optimization  may  be  achieved  in  a  way  that  is  consistent  with  Army  recruiting  interests.  For  each 
recruit,  the  count  and  proportion  of  priority  MOS  at  the  top-n  of  the  EER  list  will  be  compared  to 
the  corresponding  count  and  proportion  of  priority  MOS  at  the  top-n  of  REQUEST  list.  The 
length  n  will  be  determined  from  a  preliminary  analysis  of  REQUEST  and  EER  lists.  For  this 
analysis,  “consistent”  will  be  indicated  by  the  occurrence  of  priority  MOS  at  the  top  of  the 
EP  AS-enhanced  list. 


Classification  Efficiency  Analysis 

The  EPAS  optimization  model  is  in  part  based  on  matching  recruit  ASVAB  profiles  to 
the  aptitude  requirements  of  individual  MOS.  For  a  given  recruit,  the  optimization  guidance 
suggested  by  the  EPAS  list  would  tend  to  put  the  recruit  in  the  job  where  he  or  she  is  expected  to 
best  perform  based  on  the  ASVAB  profile.  This  EPAS  enhancement  of  REQUEST  is  anticipated 
to  yield  classification  gains  in  the  overall  Army  accession  cohort.  This  hypothesis  will  be 
examined  in  the  following  questions. 

•  How  does  EPAS-enhanced  REQUEST  compare  to  REQUEST  in  terms  of  overall 
classification  benefits? 

In  this  question  we  are  interested  in  comparing  the  overall  expected  performance  of  the 
Army  accession  cohort  under  the  current  REQUEST  and  EER  systems  of  classifying  recruits  to 
MOS.  The  index  in  this  analysis  is  recruit  mean  predicted  performance  (MPP).  For  a  specified 
evaluation  period,  recruits  will  be  assigned  to  MOS  using  classification  simulation  procedures 
that  will  be  conducted  separately  for  each  classification  system.  Recruit  predicted  performance 
will  be  calculated  and  averaged  separately  under  each  system  based  on  their  assigned  MOS. 
Comparison  of  the  MPP  calculated  under  the  two  classification  strategies  will  indicate  whether 
or  not  EPAS  enhancement  significantly  adds  to  the  classification  efficiency  of  REQUEST. 

•  Is  classification  efficiency  disproportionately  biased  towards  certain  MOS  to  the 
disadvantage  of  other  MOS? 

The  analysis  in  the  preceding  study  question  pertains  to  overall  Army  classification  gains. 
In  this  question,  we  are  concerned  with  the  equitable  distribution  of  potential  classification  gains 
across  MOS  that  is  anticipated  from  the  operation  of  EPAS-enhanced  REQUEST.  To  answer  this 
question,  the  MPPs  under  the  two  classification  systems  will  be  calculated  and  compared 
separately  by  MOS.  The  intra-MOS  classification  efficiency  differences  between  the 
classification  systems  will  be  examined,  and  MOS  that  exhibit  disproportionate  classification 
gains  will  be  noted.  This  analysis  is  not  concerned  with  the  disparity  in  the  MPP  across  MOS, 
which  is  related  to  the  respective  quality  requirements  of  MOS. 
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Accession  Requirement  Analysis 


Issues  that  are  addressed  in  the  following  study  questions  pertain  to  the  overriding  goal  in 
Army  recruiting  policy.  In  the  end,  anticipated  EPAS  classification  benefits  will  have  to  be 
achieved  while  at  the  same  time  meeting  Army  recruit  accession  requirements.  The  underlying 
issue  in  the  two  questions  listed  below  is  whether  or  not  the  person-job  matching  optimization 
process  of  the  EPAS  enhancement  adversely  affects  the  Army’s  ability  to  meet  accession  goals. 

•  How  does  EPAS-enhanced  REQUEST  compare  to  REQUEST  in  meeting  overall  Army 
monthly  accession  goals? 

The  analysis  for  this  question  will  compare  how  well  monthly  overall  Army  accession 
goals  are  met  under  the  REQUEST  and  EER  classification  strategies.  The  index  in  this  analysis 
will  be  the  percentages  of  Army  accessions  relative  to  the  monthly  target  accessions,  calculated 
separately  under  the  REQUEST  and  EPAS-enhanced  REQUEST  classification  systems.  This 
analysis  is  important  to  the  Army,  as  monthly  accession  goals  are  critical  building  blocks  in 
meeting  fiscal  year  recruiting  goals.  In  scheduling  training  class  dates  and  seat  availability 
throughout  a  given  fiscal  year.  Army  recruiting  policy  makers  take  into  account  seasonal  patterns 
of  recruit  quantity  and  quality.  To  the  extent  possible,  weekly  and  monthly  training  seats  are  not 
to  be  left  unfilled,  otherwise  they  become  lost  training  opportunities  with  associated  costs. 

•  How  does  EPAS-enhanced  REQUEST  compare  to  REQUEST  in  meeting  accession  goals  for 
priority  MOS? 

This  question  focuses  on  MOS  that  are  most  critical  to  the  Army  in  carrying  out  its 
mission.  Analyses  related  to  this  question  will  examine  the  monthly  and  total  fiscal  year 
accessions  of  each  priority  MOS  under  REQUEST  and  EPAS-enhanced  REQUEST 
classification  conditions.  For  a  specified  priority  MOS,  the  analysis  index  would  be  the 
percentage  of  total  accessions  relative  to  the  goal  for  the  MOS.  Priority  MOS  that  will  be 
considered  in  this  analysis  will  be  identified  with  the  help  of  Army  recruiting  policy  makers. 
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III.  OFF-LINE  FIELD  TEST  SIMULATION  METHOD 


A  key  requirement  of  this  study  is  to  develop  a  method  of  evaluating  the  potential 
benefits  of  EPAS-enhanced  REQUEST  that  will  not  affect  the  actual  operational  selection  and 
job  assignments  of  Army  recruits.  The  field  test  design  described  in  this  report  compares  the 
current  REQUEST  MOS  assignment  and  training  reservation  system  to  an  EER  system  using  12 
months  of  actual  data  with  an  off-line  simulation  method.  The  data  encompass  Army  recruit 
supply  and  MOS  and  training  demand  data,  the  REQUEST  list  of  MOS  vacancies  and  associated 
training  classes  generated  specifically  for  each  recruit,  and  the  particular  MOS  assignment 
decisions  made  by  the  recruits  with  assistance  from  Army  guidance  counselors. 

In  this  chapter  we  describe  the  off-line  evaluation  strategy,  which  is  based  on  a 
simulation  of  a  12-month  Army  classification  process  designed  to  retain  key  realistic 
components  of  the  Army  recruit  classification  process.  The  off-line  nature  of  the  field  test  means 
that  the  study  will  not  interfere  with  actual  Army  assignments  during  the  evaluation  period.  It 
does  place  limitations  on  inferences  about  the  realism  of  the  findings,  but  the  design  minimizes 
these  limitations. 

The  purpose  of  the  field  test  is  to  evaluate  whether  the  person-job  matching  procedures  of 
EPAS,  when  combined  with  REQUEST  classification  processes  and  subjected  to  real  world 
constraints  (e.g.,  MOS  priorities  and  short  DEP  periods),  will  improve  recruit-MOS  matching 
(measured  in  terms  of  a  job  performance  index)  and  meet  critical  MOS  and  accession 
requirements.  It  is  the  simulation  of  the  effects  of  EPAS  that  is  the  main  rationale  for  the  off-line 
aspect  of  the  field  test.  The  simulation  is  designed  to  retain  the  main  features  of  the  Army  recruit 
classification  process  subject  to  the  off-line  evaluation  constraint. 

Our  discussion  of  the  off-line  field  test  method  begins  with  a  description  of  the  four 
components  that  will  be  included  in  the  simulation  of  REQUEST  and  EPAS-enhanced 
REQUEST.  We  will  then  specify  the  three  evaluation  periods  that  can  be  included  in  the  study. 
This  will  be  followed  by  a  description  of  the  classification  simulation  itself,  including  the 
strengths  and  weaknesses.  The  next  section  will  present  in  detail  two  alternative  methods  of 
generating  REQUEST  MOS  training  lists  for  the  simulations.  This  process  is  critical  for 
obtaining  valid,  realistic  simulation  results.  The  methods  will  have  different  impacts  on  the 
results,  so  their  strengths  and  weakness  will  be  discussed.  Finally,  we  will  describe  the  MOS 
assignment  decision-making  model  that  will  be  the  tool  for  generating  off-line  MOS  training 
class  assignments.  Without  this  decision  modeling  technique  the  off-line  field  test  would  not 
produce  operationally  credible  results. 

Main  Components  of  Army  Classification  System  Included  in  Field  Test 

The  operational  Army  recruit  classification  system  can  be  divided  into  four  main 
components  that  are,  jointly  and  separately,  relevant  to  investigating  the  research  questions  in 
this  field  test.  The  first  and  second  components,  respectively,  are  the  supply  of  Army  contracts 
and  the  demand  to  fill  MOS  and  training  classes.  These  two  components  together  define  the 
Army  recruit  classification  environment.  The  third  and  fourth  components,  respectively,  are  the 
process  of  managing  the  availability  of  MOS  and  training  class  opportunities  for  individual 
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recruits  through  the  REQUEST  system,  and  the  decision-making  process  involved  in  assigning 
recruits  to  MOS,  which  is  conducted  by  Army  counselors  at  Military  Entrance  Processing 
Stations  (MEPS).  Generally,  the  Army  uses  the  REQUEST  system,  managed  by  the  U.S.  Total 
Army  Personnel  Command  (PERSCOM)  in  conjunction  with  operators  and  analysts  at  the  U.S. 
Army  Recruiting  Command  (USAREC)  and  Army  counselors  at  the  MEPS,  to  manage  the  flow 
of  recruit  supply  and  to  ensure  that  MOS  and  training  seat  demand  are  met.  Guidance  counselors 
work  with  a  recruit’s  ASVAB  test  scores  and  career  interests  to  identify  a  person-job  match  that 
balances  Army  accession  requirements  with  recruit  personal  goals. 

The  proposed  off-line  classification  simulation  method  is  a  combination  of  actual  and 
simulated  versions  of  the  preceding  four  components.  The  first  two  system  components,  which 
form  the  Army  recruit  classification  environment,  will  be  the  missioned  supply  of  recruit 
contracts  and  actual  demand  for  filling  Army  jobs,  represented  in  terms  of  MOS  and  quality 
distribution  quotas  covering  the  evaluation  period.  The  third  and  fourth  system  components, 
which  are  constrained  by  the  requirement  to  conduct  an  off-line  field  test  that  does  not  impact 
Army  recruiting,  accession  management,  and  classification,  will  be  represented  in  the  off-line 
simulation  by  empirically  derived  procedures  described  in  this  report. 

Supply:  Army  Recruit  Cohort  Data 

Evaluation  of  the  potential  benefits  of  an  EER  system  over  REQUEST  will  be  based  on 
actual  Army  recruit  cohort  data.  Recruit  data  will  pertain  to  determining  recruit  qualifications  for 
different  MOS.  This  will  include  recruit  ASVAB  test  scores,  demographics,  and  physical  and 
other  attributes  that  are  used  for  screening  purposes  in  some  MOS. 

Data  from  the  REQUEST  transaction  between  a  recruit  and  guidance  counselor  at  the 
MEPS  will  also  be  recorded.  The  two  most  important  of  these  are  the  REQUEST  MOS  class  list 
(a.k.a.  training  opportunities  list,  which  includes  information  on  enlistment  bonuses,  educational 
benefits  and  other  relevant  incentives  related  to  the  MOS  choices  available  to  the  recruit)  that  is 
rank-ordered  by  start  date,  and  the  recruit’s  actual  MOS  and  training  class  choice.2  Retaining  the 
connection  between  recruits  and  their  MEPS  transaction  dates  in  the  assignment  simulation  will 
allow  us  to  capture  the  seasonality  of  recruit  quantity  and  quality  inflow  throughout  the 
evaluation  period. 

MEPS  location  and  selected  counselor  attributes  (e.g.  gender,  length  of  service,  and 
number  of  years  as  counselor)  may  also  be  recorded.  These  additional  data  form  part  of  the 
overall  recruit  cohort  characteristics,  and  could  be  included  in  the  simulated  classification  of 
recruits  if  they  help  explain  the  pattern  of  recruit  MOS  training  choices  observed  in  the  data. 

Demand:  MOS  Vacancies  and  Training  Seat  Opportunities 

The  demand  side  of  recruit-MOS  assignments  will  be  represented  by  the  schedule  of 
MOS  training  seats  that  is  created  and  updated  by  PERSCOM  Accession  Management  Branch 
(PERSCOM/ AMB).  This  schedule  describes  the  seat  availability  for  specific  MOS  by  weekly 


2  The  Army’s  Keystone  Office,  manager  of  the  REQUEST  system,  provided  FY  2002  data  to  the  EPAS  field  test 
project  as  customized  Keyview  extracts  that  were  retrieved  on  a  weekly  basis. 
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class  start  date,  training  location,  and  gender  distribution.  The  training  seat  totals  and  allocation 
among  MOS  are  determined  from  overall  Army  and  MOS  level  FY  accession  goals.  Specified 
monthly  accession  targets  are  used  to  monitor  and  direct  filling  of  the  overall  Army  and  MOS 
level  requirements.  This  helps  to  keep  accessions  on  track  to  meet  fiscal  year  goals.  Demand  is 
also  characterized  by  recruit  quality  distribution  defined  in  terms  of  AFQT  (Armed  Forces 
Qualification  Test)  and  education  categories  at  the  overall  Army  level.  This  is  transferred  to 
variable  MOS  level  quality  distribution  goals,  depending  on  the  training  and  job  requirements  in 
each  MOS. 

The  number  of  MOS  vacancies  and  open  training  seats  in  the  field  test  will  be  updated  as 
recruits  are  assigned  off-line  to  MOS  training  class  start  dates  throughout  the  evaluation  period. 
As  in  the  real  world,  unfilled  class  seats  will  be  considered  lost  as  assignments  move  past  the 
class  start-date.  Beyond  this  simple  class-fill  update  procedure,  there  are  systematic  and 
subjective  factors  that  ultimately  determine  the  periodic  availability  of  training  classes  to 
individual  recruits.  The  way  these  factors  will  be  handled  in  the  simulation  procedure  is  related 
to  the  way  the  REQUEST  list  will  be  generated,  and  is  discussed  below  in  the  section  entitled 
“Methods  of  Generating  REQUEST  MOS  Training  Class  Lists”. 

REQUEST  and  EP AS-Enhanced  REQUEST  MOS  Training  Class  List 

As  described  above  a  guidance  counselor  works  with  a  recruit  to  select  an  MOS  and 
training  class  with  a  start  date  that  corresponds  to  the  recruit’s  availability.  The  basis  of  this 
recruit  classification  transaction  is  an  MOS  training  class  list.  REQUEST  generates  the  list  of  up 
to  35  MOS  class  start  date  combinations  for  the  counselor  to  review  with  the  recruit.3  All  MOS 
on  the  list  are  those  for  which  the  recruit  has  a  passing  Aptitude  Area  (AA)  score.  The 
REQUEST  system  ranks  the  MOS  according  to  the  Army’s  needs  to  fill  job  and  training  seat 
vacancies  during  a  particular  month.  EPAS  enhancements  to  REQUEST  are  designed  to  affect 
the  ranking  of  MOS  training  classes. 

In  contrast  to  REQUEST,  EPAS  orders  MOS  from  highest  to  lowest  AA  score  as  long  as 
they  are  above  the  cut  off  score.  EPAS  takes  into  consideration  some,  but  not  all,  of  the  training 
class  start  date  information  that  REQUEST  does.  EP AS-enhanced  REQUEST  merges  the  EPAS 
list  (called  the  EPAS  Optimal  Guidance)  with  the  REQUEST  list.  Several  merge  algorithms  are 
under  consideration.  The  most  likely  possibility  will  be  that  MOS  appearing  on  both  lists  will  be 
retained  in  the  order  determined  by  EPAS.  The  non-overlapping  MOS  on  the  REQUEST  list  will 
be  placed  at  the  bottom  in  the  order  prepared  by  REQUEST.  The  non-overlapping  MOS  in  the 
EPAS  list  will  be  dropped  from  the  merged  list. 

Recruit  MOS  Training  Class  Choice 

The  final  component  of  the  field  test  is  a  statistical  decision-making  model  of  recruit 
MOS  training  class  assignments,  which  we  are  developing  from  the  database  using  econometric 
techniques.  Model  development  depends  greatly  on  fitting  a  sample  of  about  three  months  of 
data  to  alternative  methods.  The  MOS  assignment  decision-making  model  will  play  the  part  of 


3  If  the  recruit  does  not  see  a  job  he  or  she  is  interested  in,  or  has  a  specific  job  in  mind,  then  REQUEST  also  can 
generate  a  list  of  class  dates  for  a  specific  MOS. 


9 


the  actual  transaction  between  a  recruit  and  guidance  counselor  at  the  MEPS,  which  determines 
the  recruit’s  MOS  and  training  class  choice.  It  is  the  key  component  that  permits  the  design  of  an 
off-line  evaluation  strategy,  allowing  us  to  meaningfully  recreate  recruit  MOS  training  class 
choices.  The  model  will  be  represented  by  an  equation,  from  which  the  odds  of  the  recruit 
selecting  each  training  choice  in  a  given  list  will  be  derived. 

During  the  off-line  classification  of  recruits  to  jobs,  assignment  probabilities 
corresponding  to  the  alternative  MOS  training  choices  in  the  REQUEST  or  EER  list  of  a  recruit 
will  be  calculated.  These  probabilities  will  be  a  function  of  recruit  AS  VAB  profile  and 
demographics,  the  associated  MOS  and  rank  order  training  choices  in  the  REQUEST  list,  along 
with  other  transaction  variables  that  will  be  considered,  such  as  enlistment  bonus  and  term  of 
enlistment.  The  probabilities,  in  turn,  will  determine  the  odds  of  a  recruit  “selecting”  a  particular 
training  choice  on  the  list.  This  choice  involves  a  randomization  process  that  allows  the 
possibility  of  assigning  the  “same”  recruit  to  different  training  classes  in  different  assignment 
replications.  (Refer  to  the  description  of  the  simulation  process  below.)  The  relative  frequency 
distribution  of  recruit  and  MOS  training  choice  attributes  across  many  replications  would  be 
expected  to  follow  the  choice  pattern  in  the  actual  REQUEST  transaction  data  from  which  the 
recruit  choice  model  is  derived. 

The  motivation  for  the  recruit  choice  model  approach  is  discussed  in  the  section  entitled 
MOS  Assignment  Decision-Making  Model.  A  technical  overview  is  in  Appendix  A,  including  an 
illustration  of  an  MOS  training  choice  randomization  process. 

Evaluation  Periods 

We  propose  to  analyze  the  field  test  results  using  6-  and  12-month  data.  We  can  also 
conduct  monthly  analyses.  Since  EPAS  is  a  fiscal  year  classification  optimization  model,  the  12- 
month  data  should  conform  to  a  fiscal  year  (FY)  and  cover  a  full  FY  recruit  cohort.4 

Preliminary  monthly  assessments  of  REQUEST  and  EP AS-enhanced  REQUEST  could 
be  carried  out  in  the  first  six  months  of  the  field  test.  The  main  advantage  of  one  or  more 
preliminary  analyses  is  that  they  would  provide  initial  estimates  of  the  anticipated  classification 
gains  of  EP  AS-enhanced  REQUEST.  These  estimates,  however,  would  not  reflect  any 
cumulative  classification  effects  and  would  underestimate  EPAS  benefits.5  Intermediate  analyses 
will  be  carried  out  using  a  six-month  evaluation  period.  These  results  are  expected  to  provide 
better  estimates  of  EPAS  enhancements  than  the  preliminary  analyses,  but  the  relatively  short 
evaluation  period  will  not  allow  EPAS  to  fully  capitalize  on  its  cumulative  optimization 
procedure. 


4  Twelve  months  of  data,  with  data  collection  starting  in  the  middle  of  a  fiscal  year,  could  be  used.  However,  any 
changes  in  the  recruit  cohort,  recruiting  environment  or  Army  priorities  effective  at  the  beginning  of  the  FY  may  add 
noise  in  the  database  that  could  present  some  level  of  difficulty  in  analyzing  the  results.  Further,  presenting  field  test 
findings  that  partially  cover  two  fiscal  years  to  Army  decision-makers  could  add  unnecessary  complications  to  an 
already  complex  discussion. 

5  Note  that  the  EPAS  person-job  matching  algorithm  successively  accumulates  classification  gains  monthly 
throughout  the  FY  by  optimizing  the  assignment  of  current  contractees  to  current  and  future  MOS-class-months 
through  the  use  of  monthly  supply  and  demand  projections. 
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The  final  set  of  analyses  will  most  likely  be  carried  out  for  the  full  fiscal  year  evaluation 
period.  The  REQUEST  MOS  training  list  can  be  either  the  operational  list  or  a  simulated 
version  that  will  make  the  REQUEST  and  EER  conditions  comparable.  This  is  discussed  in  the 
section  entitled  “Methods  of  Generating  REQUEST  MOS  Training  Class  Lists”.  If  the 
REQUEST  MOS  training  list  is  simulated  off-line  under  the  same  conditions  as  the  EPAS  list  for 
both  the  REQUEST  and  EER  conditions,  then  the  results  will  reflect  the  unique  fill  rate  patterns 
of  each  system,  and  allow  EPAS  to  build  upon  its  optimization  without  constraint.  We  expect  the 
12-month  analyses,  especially  if  the  REQUEST  list  is  simulated,  to  provide  more  accurate 
information  on  EPAS-related  cumulative  classification  benefits  than  the  intermediate  analysis. 

Below  we  describe  a  single  replicate  of  the  off-line  classification  simulation  process 
conducted  for  a  non-specific  evaluation  period  and  simulation  condition. 

Off-line  Classification  System  Simulation  Process 

The  four  components  needed  to  conduct  the  off-line  field  test,  which  are  described  above, 
will  be  integrated  in  a  simulation,  with  replication,  of  the  Army’s  recruit  classification  system. 
Replicated  simulations  of  the  Army  recruit-MOS  classification  process  will  be  carried  out 
separately  under  REQUEST  and  EER  conditions  and  will  generate  the  data  with  which  the 
research  questions  will  be  answered.  The  simulations  under  the  two  conditions  will  differ  in  the 
assignment  opportunities  included  in  the  MOS  training  lists  that  are  “presented”  to  recruits,  since 
the  differences  in  the  lists  mark  the  primary  research  conditions.  The  list  for  EP AS-enhanced 
REQUEST  will  be  generated  partly  off-line.  A  decision  about  whether  to  generate  the 
REQUEST  list  off-line  in  both  systems  to  maintain  comparability  of  the  conditions  must  be 
made  before  the  simulation  programs  are  developed.  As  we  mentioned  above,  this  issue  is 
discussed  in  detail  in  Methods  of  Generating  REQUEST  MOS  Training  Class  Lists. 

Figures  1  and  2  in  the  Introduction  present  graphical  representations  of  operational 
REQUEST  and  proposed  EPAS-enhanced  REQUEST,  without  including  the  modifications 
necessary  for  the  off-line  simulations.  Modified  figures  are  presented  later  in  conjunction  with  more 
detailed  descriptions  of  the  simulations.  A  single  replication  in  the  simulations  will  correspond  to  a 
complete  classification  of  all  recruit  contracts  in  the  evaluation  period  under  consideration.  Starting 
from  the  earliest  contract  in  the  evaluation  period,  recruits  will  be  classified  into  jobs  following  the 
order  of  actual  recruit  inflow  to  ensure  a  realistic  distribution  of  recruit  attributes  by  contract  date. 

At  the  start  of  each  replication  the  training  class  seats  will  be  set  equal  to  their  actual, 
respective  fills  at  the  beginning  of  the  evaluation  period  as  recorded  in  the  auxiliary  REQUEST 
data  collected  for  the  field  test.  A  list  of  MOS  training  opportunities  will  be  constructed  for  the 
recruit.  This  list  will  be  either  a  REQUEST  or  EER  list,  depending  on  the  classification  system 
condition.6  Given  the  MOS  training  opportunities  in  the  list,  the  recruit  will  be  assigned 


6  The  REQUEST  list  used  independently  in  REQUEST  itself,  or  merged  with  the  EOG  in  EPAS-enhanced 
REQUEST,  may  be  generated  on-  or  off-line  according  to  the  evaluation  period,  if  a  decision  is  made  to  simulate  the 
REQUEST  MOS  training  list.  Otherwise  the  REQUEST  list  will  be  generated  on-line.  The  EOG  will  always  be 
generated  off-line. 
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stochastically  to  a  MOS  training  class  seat  using  the  recruit  MOS  assignment  decision-making 
model  outlined  above. 

MOS  vacancies  and  class  seat  fills  will  be  updated  after  classifying  each  recruit  during 
the  evaluation  period.  These  updated  class  fills  will  be  used  as  input  in  generating  the  REQUEST 
list  of  MOS  training  opportunities  for  subsequent  recruits,  and  will  depend  on  the  classification 
system  condition  (REQUEST  or  EP AS-enhanced  REQUEST)  and  the  REQUEST  MOS  list 
generation  method  described  below.  For  the  EER  simulation,  the  updated  class  fills  at  the  start  of 
each  week  also  will  be  provided  as  input  to  the  EPAS  optimization  routine.  After  classifying  the 
last  recruit  in  each  replication,  the  training  class  seats  again  will  be  reset  to  their  actual 
REQUEST  fills  at  the  beginning  of  the  evaluation  period  before  starting  the  next  replication 
using  the  same  data.  A  single  classification  simulation  replication  will  be  completed  after  the 
recruit  with  the  last  contract  date  in  the  evaluation  period  under  consideration  is  classified. 
Repeating  the  above  replication  thirty  times  will  generate  the  complete  set  of  simulation  data  for 
a  given  condition  and  evaluation  period. 

Analysis  indices  for  the  three  sets  of  research  questions  will  be  computed  for  each 
classification  system  replication  under  REQUEST  and  EER  conditions  in  an  evaluation  period. 
For  analysis  related  to  the  job  list  comparison  research  questions,  indices  for  comparing  MOS 
training  opportunities  under  the  two  conditions  will  be  calculated  at  the  individual  recruit  level. 
Classification  efficiency  and  accession  requirements  indices  will  be  calculated  based  on  the  off¬ 
line  MOS  “assignments”  of  recruits  for  the  entire  cohort  in  the  evaluation  period.  Note  that  even 
though  the  classification  simulations  in  the  two  conditions  are  separate,  the  order  of  recruit 
contract  date  and  demand  for  MOS  training  class  seats  during  the  evaluation  period  are  fixed. 
Fixing  supply  and  demand  will  help  provide  the  basis  for  establishing  true  differences  between 
REQUEST  and  EPAS-enhanced  REQUEST,  if  they  exist. 

We  conclude  the  description  of  the  off-line  simulation  process  by  taking  note  of  the 
seemingly  awkward  scenario  of  an  individual  recruit  probably  “choosing”  different  MOS  or 
training  classes  in  different  replications.  The  key  to  understanding  the  analytical  intent  of  the 
simulation  design  is  to  disassociate  a  recruit's  actual  identity  from  his  or  her  recruit  attributes. 
The  following  two  assumptions  underlie  the  off-line  classification  simulation  strategy  in  this 
field  test.  The  first  is  that  the  seasonal  distribution  of  Army  recruit  attributes  by  MOS  contract 
date  in  the  operational  REQUEST  data  to  be  used  in  the  field  test  represents  the  pattern  typically 
observed  across  fiscal  years.  The  second  assumption  is  that  recruit  attribute  by  MOS  training 
opportunity  patterns  are  to  some  degree  revealed  in  the  field  test  data.  In  light  of  these  two 
assumptions,  the  averages  of  different  indices  across  replications  in  the  analysis  should  be 
interpreted  as  statistical  measures  of  specified  characteristics  of  a  classification  system  having 
approximately  the  same  recruit  supply  and  Army  demand  environments  as  represented  by  the 
seasonal  distribution  of  recruit  attributes  and  MOS  training  opportunities  in  the  operational  data. 

Strengths  and  Weaknesses  of  the  Off  dine  Simulation  Method  of  the  Field  Test 

The  primary  practical  strengths  of  the  off-line  simulation  method  (proposed  to  compare 
the  performance  of  EPAS-enhanced  REQUEST  to  that  of  REQUEST)  are  its  ability  to  evaluate 
EPAS  enhancements  without  interfering  in  the  operations  of  REQUEST,  impacting  the 
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attainment  of  Army  accession  missions,  disrupting  the  MEPS  recruit  counseling  process  or 
reducing  present  levels  of  recruit  training  and  job  performance.  In  addition  to  the  practical 
benefits  there  are  four  scientific  advantages  to  conducting  an  off-line  field  test. 

First,  the  off-line  simulation  method  allows  us  to  use  data  from  a  full  FY  recruit  cohort  in 
all  field  test  conditions  instead  of  using  data  sampling  techniques.  On-line  sampling  of 
REQUEST  and  EP AS-enhanced  REQUEST  would  require  temporal  and  probably  geographic 
sampling  that  would  introduce  error  variance,  which  may  be  difficult  to  detect  even  with  the  use 
of  control  variables. 

Second,  assuming  that  it  would  be  operationally  feasible  to  randomly  assign  recruits  to 
either  REQUEST  or  EER  conditions  in  an  on-line  field  test,  it  would  be  difficult,  if  not 
impossible,  to  independently  measure  the  Army-wide  effects  of  the  two  conditions.  This  is 
because  of  the  different  dynamic  feedback  process  in  the  classification  systems,  in  which 
assignments  of  recruits  to  training  classes  at  a  given  time  impact  assignments  at  a  later  time 
through  the  availability  of  class  seats.  The  use  of  sampling  approaches  for  running  REQUEST 
and  EP  AS-enhanced  REQUEST  on-line  during  a  FY  would  confound  the  separate  effects  of 
each  system  on  MOS  and  training  class  fill  rates.  Further,  developing  an  error  term  for  the 
indices  associated  with  each  set  of  research  questions  would  be  difficult  and  the  error  probably 
would  be  quite  large.  If  an  operationally  and  scientifically  acceptable  procedure  were  developed, 
it  would  be  complex  and  expensive  to  implement.  We  believe  the  off-line  simulation,  based  on 
fixed  Army  supply  and  demand  data  in  both  conditions  is  an  effective  strategy  for  controlling 
time-related  fill  rates. 

Third,  replicated  analysis  provides  an  estimate  of  the  error  variances  of  the  evaluation 
indices.  This  is  needed  for  building  confidence  bounds,  which  are  necessary  for  proper 
comparison  of  results  from  the  two  classification  system  conditions. 

Fourth,  by  using  off-line  simulations,  alternative  elements  of  the  Army  classification 
system  not  currently  in  use  may  be  considered  in  the  field  test  without  undermining  current 
recruiting  goals  or  assignments.  For  example,  different  classification  composites  and  job  families 
could  be  examined.  The  length  of  the  Delayed  Entry  Program  (DEP)  could  be  manipulated  to 
study  potential  benefits  related  to  EP  AS-enhanced  REQUEST  or  REQUEST.  (Note  that  EPAS 
is  designed  to  use  DEP  length  to  create  efficiencies).  These  types  of  analyses,  especially  the 
DEP  analysis,  could  be  highly  valuable  to  Army  policymakers  for  classification  strategy  and 
planning  purposes. 

On  the  other  side  of  the  equation  are  the  weaknesses  of  the  off-line  simulation  method  of 
the  proposed  field  test.  First,  field  test  results  from  a  simulation  study,  which  cannot  include  all 
operational  factors,  may  be  considered  less  credible  than  operational  field  tests  by  policymakers 
and  managers.  This  is  not  necessarily  true  as  implied  by  the  discussion  of  strengths  above.  On  a 
more  general  level,  a  well-designed  simulation  study  may  produce  more  accurate  findings 
because  of  the  greater  control  of  irrelevant  error  variance. 

A  second  weakness  is  that  off-line  simulation  methods  are  challenging  and  time 
consuming,  especially  when  they  involve  complex  systems  such  as  REQUEST  and  EPAS.  In 
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particular,  results  will  only  be  valid  to  the  extent  that  analytical  techniques  that  replace 
subjective  operational  procedures  are  meaningful  and  reliable.  An  example  of  such  an  analytical 
technique  is  the  recruit  MOS  assignment  decision-making  model  that  will  play  the  part  of  the 
recruit  and  counselor  assignment  transaction  at  the  MEPS.  This  model  is  a  stochastic 
representation  of  the  assignments  based  on  recruit  attributes  and  the  Army  accession 
environment  during  the  evaluation  period.  The  study  cohort  will  be  considered  a  representative 
sample  of  the  current  Army  accession  population. 

Methods  of  Generating  REQUEST  MOS  Training  Lists 

The  availability  of  MOS  training  opportunities  to  individual  recruits  at  the  MEPS  are 
generally  driven  by  recruit  qualifications,  MOS  vacancies,  the  updated  training  seat  schedule, 
accession  requirements,  and  quality  distribution  goals  within  and  across  MOS.  Open  MOS  for 
which  a  recruit  is  qualified  are  ranked  according  to  Army  recruiting  and  fill  priorities  by 
REQUEST.  The  overriding  objective  of  managing  the  availability  of  class  seats  is  the  efficient 
use  of  training  resources  in  meeting  Army  accession  goals. 

REQUEST  employs  an  elaborate  procedure  for  opening  and  ranking  MOS  training  class 
dates,  incorporating  systematic  and  subjective  factors  in  the  process.  The  final  output  of  this 
process  is  a  rank  ordered  list  of  MOS  training  class  start  dates  that  reflects  recruit  characteristics 
and  prevailing  Army  priorities  at  the  time  of  the  MEPS  transaction.  Recruits  with  exactly  the 
same  characteristics,  for  example,  may  obtain  different  MOS  training  lists  depending  on  their 
signing  dates,  real  time  changes  in  individual  MOS  fill  rates,  and  shifts  in  Army  priorities. 

Two  approaches  could  be  employed  in  this  field  test  to  construct  the  REQUEST  MOS 
class  start  date  list  of  a  recruit.  The  first  procedure  is  primarily  based  on  the  recruit’s  actual 
REQUEST  list,  while  the  second  procedure  would  be  based  on  simulation  procedures  that  would 
approximate  key  mechanisms  in  REQUEST.  These  determine  availability  and  rank  ordering  of 
MOS  training  class  start  dates.  We  describe  below  the  two  REQUEST  MOS  training  list 
generation  methods,  discuss  for  which  evaluation  periods  they  are  appropriate,  and  conclude  by 
comparing  their  respective  utilities  for  the  field  test. 

Fixed  Method  of  Generating  the  REQUEST  Lists 

The  first  approach  to  constructing  MOS  training  class  lists  under  REQUEST  and  EER 
classification  conditions  relies  on  the  actual  training  class  opportunities  presented  to  recruits  at  the 
MEPS,  which  are  contained  in  the  REQUEST  transaction  database.  This  simple  approach  allows  us 
to  circumvent  simulating  complex  systematic  and  subjective  factors  used  by  REQUEST  to  create 
lists  of  MOS  training  classes,  with  rank  ordered  class  start  dates  that  reflect  Army  priorities  and 
training  schedules.  We  refer  to  this  as  the  fixed  method  of  list  construction.  Below  we  describe  the 
approach  under  each  classification  simulation  condition. 

REQUEST  Classification  Simulation  Condition 

Figure  3  depicts  the  off-line  simulation  of  REQUEST  based  on  the  fixed  method  of 
generating  the  MOS  training  class  list.  The  operational  REQUEST  PJM  process  is  represented 
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by  the  black  box  in  the  center  of  the  diagram.  The  strategy  in  the  fixed  method  would  be  to 
access  directly  the  REQUEST  transaction  data,  which  includes  the  list  shown  to  each  recruit.  As 
mentioned  above,  this  would  allow  us  to  bypass  the  operational  inputs  shown  to  the  left  of 
REQUEST,  as  well  as  the  REQUEST  process  itself.  The  REQUEST  list  is  shown  as  the  box  to 
the  right  of  the  black  box  PJM  process,  and  is  referred  to  as  the  fixed  actual  list  to  indicate  that  it 
is  operational  data. 


Figure  3.  Graphical  Depiction  of  the  Off-Line  Simulation  of  the  REQUEST  Classification 
System  Based  on  the  Fixed  Method  of  Generating  the  REQUEST  List 


The  fixed  REQUEST  list  would  produce  an  operational  MOS  and  training  seat  fill  pattern 
at  the  actual  contract  date  of  a  recruit  that  is  somewhat  different  from  the  pattern  in  the 
simulation  at  the  same  contract  date.  This  is  because  the  recruit  MOS  assignment  decision¬ 
making  model  in  the  classification  simulations  provides  random  variation  in  recruit  MOS 
training  class  choices  (indicated  by  the  oval  on  the  right  side  of  Figure  3).  A  simple  adjustment 
would  be  made  to  the  actual  REQUEST  list  by  dropping  MOS  training  classes  that  are  filled  at 
the  recruit’s  contract  date  in  the  simulation.  MOS  training  opportunities  that  remain  according  to 
the  simulated  classification  process  will  form  the  recruit’s  off-line  adjusted  REQUEST  list, 
which  appears  to  the  right  of  the  fixed  actual  list. 

The  upper  and  lower  feedback  loops  in  Figure  3  show  how  the  operational  REQUEST 
data  and  the  results  of  the  simulated  recruit  MOS  choices  update  the  operational  and  simulated 
classification  systems,  respectively.  The  recruit  assignment  choice  in  the  simulation  is  fed  back 
(upper  loop)  to  an  off-line  database  that  contains  continuously  updated  MOS  and  training  seat 
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vacancies.  These  updates  are  used  to  adjust  the  fixed  actual  list.  The  lower  loop  indicates  that 
actual  recruit  choice,  which  is  based  on  the  fixed  list,  is  fed  back  into  the  operational  REQUEST 
system  by  updating  actual  MOS  and  training  seat  vacancies.  The  operational  updates  become 
REQUEST  inputs  that  impact  subsequent  fixed  lists. 

At  the  beginning  of  the  evaluation  period,  the  difference  in  operational  and  simulated  fill 
patterns  is  expected  to  be  relatively  small,  as  simulation  and  actual  assignments  will  start  with 
the  same  initial  schedule,  and  then  becomes  larger  as  subsequent  assignments  are  made. 
However,  the  discrepancy  between  actual  and  off-line  fill  patterns  will  be  restricted  for  the 
following  reasons.  First,  through  currently  tight  DEP  management  control  mechanisms,  the 
window  of  available  class  start  dates  in  a  REQUEST  list  is  typically  confined  to  a  few  weeks 
from  the  signing  date  of  a  recruit  (except  for  high  school  seniors).  Second,  in  the  off-line 
assignments  of  recruits,  adjustments  will  mostly  be  made  by  dropping  filled  training  classes  from 
the  typically  large  lists  of  class  start  dates  associated  with  MOS  on  the  original  list.  The 
combination  of  these  two  factors  will  have  the  desired  effect  of  keeping  the  off-line,  simulated, 
recruit  assignment  pattern  close  to  that  which  actually  will  be  realized  during  the  evaluation 
period. 


EP 'AS-Enhanced  REQUEST  Classification  Simulation  Condition 

Figure  4  is  a  diagram  of  the  EER  simulation  based  on  the  fixed  method  of  generating  the 
REQUEST  list.  Examination  of  the  process  shows  that  EPAS  and  REQUEST  will  obtain  MOS 
and  training  seat  vacancy  updates  (shown  on  the  left  side  of  the  figure)  from  two  different 
sources,  unlike  the  proposed  operational  system  that  would  use  the  same  source.  If  the  fixed 
method  is  used  in  the  simulations,  a  separate  off-line  database  of  continuously  updated  fill 
patterns  will  be  set  up  as  an  input  for  EPAS  through  the  contract  mission  database  and  the  PJM 
process,  which  will  be  run  off-line  because  it  is  not  operational.  Since  the  fixed  method  uses 
operational  REQUEST,  actual  MOS  and  training  seat  fill  patterns  will  be  inputs  to  REQUEST. 
The  operational  list  is  relabeled  the  fixed  actual  REQUEST  list  in  the  simulations  because  the  list 
for  a  recruit  becomes  a  fixed  data  point. 

As  in  the  REQUEST  classification  simulation  based  on  the  fixed  method,  EPAS- 
enhanced  REQUEST  would  use  the  simulated  MOS  and  training  seat  vacancies  to  create  an  off¬ 
line  adjusted  REQUEST  list,  in  which  MOS  training  opportunities  that  were  filled  to  capacity  in 
the  simulation  process  would  be  dropped.  However,  the  process  of  creating  the  list  does  not  stop 
there.  The  adjusted  REQUEST  list  is  merged  with  the  EPAS  optimal  guidance  (EOG)  that 
reorders  it  using  EPAS-REQUEST  merge  rules.  Simulated  recruit  choices  are  made  with  the 
MOS  assignment  decision-making  model.  The  results  are  fed  back  to  the  off-line  MOS  and 
training  seat  vacancy  database  used  to  adjust  the  REQUEST  list  so  that  it  accounts  for  previous 
off-line  assignments  and  as  an  input  to  EPAS. 

If  EPAS  reordering  of  MOS  training  classes  is  significant,  we  could  observe  a  substantial 
difference  between  the  fill  patterns  from  off-line  simulated  classification  under  fixed  EER  and 
fixed  REQUEST  simulation  conditions.  This  difference,  however,  will  likely  be  limited  because 
the  lists  used  for  off-line  classification  under  the  two  conditions  originate  from  the  same  actual 
REQUEST  MOS  training  list.  A  similar  constraint  would  occur  in  an  operational  EER  system, 


16 


because  currently  tight  DEP  management  controls  would  keep  the  REQUEST  and  EER  fill 
patterns  from  diverging  too  much. 


We  describe  in  this  section  a  second  method  for  generating  the  REQUEST  MOS  training 
class  lists  for  the  REQUEST  and  EER  classification  system  simulations.  This  procedure  is 
designed  to  circumvent  the  weaknesses  of  the  fixed  method  described  above  (i.e.,  the 
underestimation  of  cumulative  effects).  The  fixed  method  would  use  the  actual  monthly  MOS 
and  training  seat  requirements  and  vacancies  (i.e.,  the  Army’s  actual  demand  data).  These  data 
would  be  based  on  monthly  operational  REQUEST  assignments  and  adjustments  to  Army  MOS 
training  requirements  made  during  the  evaluation  period.  The  dynamic  simulation  method  of 
generating  MOS  training  class  lists  involves  synthetically  constructing  off-line  the  REQUEST 
sequential  person-job  matching  process  in  both  the  REQUEST  and  EER  classification 
conditions.  The  significant  challenges  of  developing  this  procedure  are  outlined  in  the  next 
section. 


Challenges  of  Developing  the  Dynamic  Simulation  Method  of  Generating  the  REQUEST  List 


Devising  a  method  for  constructing  a  dynamically  simulated  REQUEST  list  that  will 
accurately  represent  operational  REQUEST  in  the  off-line  field  test  is  a  complex  and  difficult 
task.  The  method  should  account  for  periodic  changes  in  numerous  factors  such  as  the  ranking  of 
priority  MOS,  and  the  intricate  DEP  management  process  employed  by  PERSCOM  to  control 
MOS  training  class  seat  availability  over  time.  While  both  priority  MOS  ranking  and  DEP 
controls,  for  example,  have  rational  bases  that  support  Army  recruiting  goals,  they  also  involve 
partly  subjective  decisions. 

The  goal  of  the  dynamic  method  is  not  the  complete  off-line  reproduction  of  procedures 
underlying  operational  REQUEST.  This  would  be  impossible  because  the  subjective  components 
of  the  REQUEST  PJM  process  cannot  be  adequately  modeled.  The  goal  would  be  to  develop  a 
model  of  REQUEST  that  meets  a  predetermined  standard  in  samples  of  recruits  taken  from  the 
field  test  database.  The  two  key  features  that  must  be  modeled  validly  and  reliably  in  the 
dynamic  method  are  management  of  the  DEP  length  and  the  ranking  of  priority  MOS,  which 
reflect  variable  Army  accession  and  classification  goals. 

Another  feature  of  REQUEST  that  could  be  modeled  is  the  periodic  update  of  the  training 
class  schedule,  which  adjusts  for  DEP  loss,  progress  in  meeting  recruiting  goals,  and  possible 
changes  in  recruiting  policy.  Modeling  of  operational  updates  should  be  done  in  conjunction  with 
analyzing  why  and  how  update  decisions  are  made  so  that  the  time  element  is  captured 
accurately  in  the  off-line  REQUEST  simulation.  Other  factors  that  impact  the  REQUEST  PJM 
process  through  the  availability  of  training  seats  are  the  sharing  of  MOS  quotas  for  the  Regular 
Army,  National  Guard,  and  Reserves,  and  the  relaxing  of  MOS  quotas  for  female  recruits,  as 
assignments  are  made  close  to  class  start  dates. 

The  rational  elements  that  determine  important  features  of  the  REQUEST  PJM  process 
could  be  accounted  for  analytically  more  or  less  depending  on  the  amount  and  quality  of 
available  information,  but  the  task  would  be  time  consuming.  Using  MOS  priority  ranking  as  an 
example,  the  rational  elements  would  be  represented  by  the  algorithm  of  approximately  10 
weighted  operational  variables  used  by  PERSCOM  to  determine  a  preliminary  ranking.  The 
variance  accounted  for  by  the  subjective  elements  could  be  partially  estimated  by  comparing  the 
preliminary  order  to  the  PERSCOM  actual  ranking.  Similar  techniques  could  be  used  with  the 
other  important  features  of  REQUEST,  but  there  is  no  way  to  know  in  advance  how  realistic  the 
dynamic  method  would  be  in  simulating  the  MOS  training  class  list. 

REQUEST  Classification  Simulation  Condition 

If  the  dynamic  method  is  used  in  the  field  test  to  generate  the  REQUEST  list  off-line,  the 
MOS  training  class  opportunities  would  be  dynamically  simulated  in  turn  for  each  recruit,  by 
order  of  their  actual  contract  date,  under  the  REQUEST  simulation  condition.  Overall,  the 
construction  of  the  dynamic  REQUEST  list  will  be  a  two-step  process.  The  initial  step  will  be 
determining  the  composition  of  the  list,  which  will  involve  mechanisms  for  screening  MOS  and 
training  start  dates,  based  on  recruit  qualification,  DEP  management,  and  training  class 
scheduling  factors.  The  eligibility  of  a  recruit  for  different  MOS  training  options  will  be 
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determined  primarily  by  AA  cut  scores  and  gender  restrictions.  Recruit  physical  attributes  and 
security  requirements  should  be  included  in  the  simulated  model  of  the  REQUEST  PJM  process 
for  certain  MOS.  After  the  composition  of  the  list  is  finalized,  the  next  step  is  to  rank  order  the 
training  opportunities  in  the  list  to  reflect  Army  recruiting  priorities.  All  information  required  in 
this  two-step,  simulated  REQUEST  list  construction  process  will  be  based  on  dynamically 
updated  MOS  and  training  class  fill  rates  associated  with  the  actual  recruit  and  his  or  her  contract 
date  in  the  off-line  simulation  environment. 

Figure  5  graphically  demonstrates  how  the  dynamic  simulation  method  of  generating  the 
REQUEST  list  would  be  carried  out  in  the  field  test. 7  As  we  stated  above,  the  basis  of  this 
method  is  a  model  of  the  REQUEST  PJM  process  that  would  be  developed  as  part  of  the  field 
test.  The  black  box  in  the  center  of  Figure  5  represents  simulated  REQUEST.  It  would  be 
developed  using  the  facets  of  REQUEST  that  are  most  important  in  determining  the  composition 
and  rank  ordering  of  the  MOS  training  class  list.  These  include  external,  operational  factors 
represented  by  two  of  the  three  icons  to  the  left,  and  feeding  into  the  REQUEST  model. 


Figure  5.  Graphical  Depiction  of  the  Off-Line  Simulation  of  the  REQUEST  Classification 
System  Based  on  the  Dynamic  Simulation  Method  of  Generating  the  REQUEST  List 


Looking  at  the  top  left  of  Figure  5,  the  first  database  consists  of  the  actual,  operational 
MOS  and  training  requirements  established  by  Army  policy  makers  for  the  FY  evaluation  period. 
The  second  database  consists  of  the  continuously  updated  fill  rates  for  MOS  and  training  class 
seats.  The  off-line  simulation  process  used  in  this  field  test  is  a  technique  for  comparing 
REQUEST  and  EP AS-enhanced  REQUEST  under  realistic  conditions  using  actual  recruit  supply 


7  Note  that  operational  REQUEST  is  a  dynamic  system  in  the  sense  that  it  is  an  iterative,  feedback  process.  We  use 
simulation  to  refer  to  both  the  modeling  of  REQUEST  (i.e.,  it  simulates,  or  is  a  synthetic  version  of,  operational 
REQUEST)  and  that  the  overarching  field  test  strategy  is  an  off-line  simulation  of  the  Army’s  classification  process,  a 
major  component  of  which  is  REQUEST.  Although  the  dynamic  simulation  method  of  generating  the  REQUEST  list 
results  in  a  purely  off-line,  simulation  field  test  (while  the  fixed  method  relies  on  operationally  produced  REQUEST 
lists,  which  become  part  of  the  off-line  simulation  process),  actual  recruit  supply,  and  MOS  and  training  demand,  data 
are  inputs  into  the  REQUEST  model.  The  operational  data  and  accuracy  of  the  REQUEST  PJM  model  make 
independent  contributions  to  the  fidelity  of  this  approach  to  the  field  test  simulations. 
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and  MOS  and  training  class  demand  data.  Updating  MOS  and  training  vacancy  data  occurs  off¬ 
line  with  both  the  fixed  and  dynamically  simulated  REQUEST  list  generation  method,  and  is  a 
direct  function  of  recruit  MOS  choices  (shown  on  the  right  of  Figure  5),  which  are  modeled  in 
the  simulation  by  the  recruit  MOS  assignment  decision-making  model.  The  dynamic  method 
would  simulate  this  process  solely  on  the  basis  of  the  off-line  simulated  assignments.  In  contrast, 
the  fixed  method  uses  an  off-line  adjusted  REQUEST  list  formed  by  accessing  the  recruit’s 
actual  REQUEST  MOS  training  class  list  and  modifying  it  to  reflect  the  simulated  MOS  and 
training  fill  rates. 

The  “Other  Operational”  icon  in  Figure  5  that  feeds  data  to  REQUEST  represents  all 
other  Army  considerations,  managed  by  U.S.  Total  Army  Personnel  Command  (PERSCOM)  and 
U.S.  Army  Recruiting  Command  (USAREC),  that  impact  the  REQUEST  MOS  training  class  list 
shown  to  recruits.  Only  the  most  important  of  these  (e.g.,  the  DEP  management  process  and 
priority  MOS)  would  be  included  in  modeling  REQUEST.  The  third  icon,  actual  recruit 
qualifications  and  window  of  availability,  refers  to  the  recruit  database,  which  includes  contract 
start  date,  for  the  evaluation  period. 

In  summary,  the  process  of  developing  the  simulated  REQUEST  PJM  model  would 
combine  modeling  operational  recruiting,  assignment,  and  accession  processes,  which  will 
approximate  the  operational  REQUEST  system  more  or  less  accurately,  with  actual  recruit 
characteristics  data  and  operational  MOS  and  training  class  requirements  data.  Continuously 
updated  MOS  and  training  seat  vacancy  data  are  also  part  of  the  model.  At  the  start  of  the 
simulation  these  data  would  be  set  to  the  operational  Army  values  and  the  simulated  REQUEST 
model  would  produce  what  we  refer  to  as  a  dynamically  simulated  REQUEST  MOS  training 
class  list.  The  data  record  for  the  recruit  with  earliest  contract  date  in  the  database  would  serve  as 
the  first  recruit,  and  the  dynamically  simulated  REQUEST  list  would  reflect  his  or  her 
qualifications.  The  assignment  decision-making  model  is  applied  to  the  simulated  list  to  simulate 
the  recruit  MOS  training  class  choice.  This  information  would  be  fed  back  to  the  MOS  and 
training  seat  vacancies  database,  which  would  be  updated  according  to  the  simulated  assignment. 
This  would  start  the  second  iteration  of  the  classification  simulation  process,  this  time  with 
dynamically  simulated  MOS  and  training  seat  vacancies.  The  process  would  continue  until  the 
data  record  for  the  recruit  with  the  last  contract  date  was  assigned.  This  would  form  1  of  the  30 
classification  system  replications  that  make  up  one  simulation  condition. 

EP AS-Enhanced  REQUEST  Classification  Simulation  Condition 

In  the  EP AS-enhanced  REQUEST  simulation  condition  the  dynamically  simulated 
REQUEST  list  of  MOS  training  opportunities  would  be  constructed  exactly  as  in  the  REQUEST 
condition  described  above.  As  shown  in  Figure  6  the  difference  is  that  the  EPAS  optimal  guidance 
list  of  MOS  would  be  merged  with  the  dynamically  simulated  REQUEST  list.  The  resulting 
dynamically  simulated  EER  list  would  be  rank  ordered  following  the  same  merge  rules  as  those  of 
the  proposed  operational  system.  The  procedure  for  simulating  a  recruit’s  MOS  training  class 
choices  also  would  be  the  same  as  in  the  REQUEST  condition. 

Figure  6  shows  that  the  result  would  be  fed  back  to  update  the  MOS  and  training  class 
vacancies.  The  updates  would  be  input  into  both  REQUEST  and  EPAS  PJM  processes.  The 
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dynamic  method  would  create  EER  lists  that  reflect  fill  rates  from  purely  simulated  recruit  MOS 
training  choices.  _ 
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Figure  6.  Graphical  Depiction  of  the  Off-Line  Simulation  of  the  EPAS  -enhanced  REQUEST 
Classification  System  Based  on  the  Dynamic  Method  of  Generating  the  REQUEST  List 


Strengths  and  Weaknesses  of  Fixed  and  Dynamic  Simulation  Methods  of  Generating  the 
REQUEST  List 

Irrespective  of  the  method  of  generating  the  REQUEST  list,  the  off-line  classification 
simulations  of  both  REQUEST  and  EPAS -enhanced  REQUEST  would  not  be  as  accurate  as 
conducting  a  fully  operational  evaluation  under  optimal  conditions.  However,  a  successful 
operational  evaluation  would  be  difficult,  if  not  impossible,  and  expensive  to  conduct.  The  major 
drawbacks  to  an  operational  field  test  are  constraints  related  to  disrupting  recruiting,  accession 
management,  and  assignment  procedures,  and  the  difficulties  of  devising  and  implementing 
strategies  for  obtaining  adequate,  independent  samples  of  assignments  made  with  the  two 
systems. 

The  strength  of  the  dynamic  simulation  method  of  generating  the  REQUEST  lists  is  that 
it  provides  the  basis  for  conducting  the  classification  simulations  of  both  REQUEST  and  EPAS- 
enhanced  REQUEST  completely  off-line  in  a  manner  that  closely  imitates  the  operational 
processes.  In  contrast,  the  fixed  method  of  generating  the  REQUEST  lists  accesses  the  actual 
(i.e.,  fixed  in  the  database)  MOS  training  class  lists  shown  to  the  recruits.  The  lists  then  are 
adjusted  to  drop  any  MOS  training  classes  that  were  filled  to  capacity  by  preceding  assignments 
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in  the  simulations.  The  adjustment  adds  a  source  of  error  to  the  simulations  that  is  not  present 
with  the  dynamic  method. 

The  advantage  of  the  dynamic  list  generation  method  will  be  small  in  the  first  few 
months  of  the  evaluation  period  because  the  differences  in  the  MOS  and  training  class  vacancy 
patterns  between  the  fixed  list,  which  is  taken  from  operational  REQUEST,  and  the  dynamically 
simulated  list  will  be  small.  The  differences  will  increase  over  time,  resulting  in  more 
adjustments  of  the  lists  to  account  for  the  simulated  fill  patterns  that  occur  with  the  fixed  method. 
We  would  expect  the  fixed  method  of  generating  the  REQUEST  list  to  be  adequate  for  the 
REQUEST  condition  for  the  6-month  evaluation  period,  while  the  dynamic  method  is  preferable 
for  the  12-month  field  test.  As  we  discuss  below,  the  dynamic  approach  would  be  best  for 
evaluating  EP AS-enhanced  REQUEST  in  both  evaluation  periods. 

The  dynamic  method  of  generating  REQUEST  lists  will  be  most  beneficial  for  evaluating 
the  EER  system.  This  is  because  the  EPAS  optimization  procedure  places  a  high  weight  on  the 
match  of  recruits  to  MOS  in  which  they  are  expected  to  perform  well  according  to  their  AA 
scores.  This  effect  provides  cumulative  benefits  by  yielding  MOS  and  training  seat  vacancy 
patterns  that  would  be  favorable  for  subsequent  recruit  assignments  in  terms  of  person-job  fit. 
Naturally,  this  effect  will  be  larger  if  lists  are  created  dynamically  as  EPAS  would  not  be 
constrained  by  the  fixed  actual  REQUEST  MOS  and  training  class  list.  The  EPAS  feature  of 
using  contract  mission  “forecasts”  in  the  optimization  procedure  to  facilitate  optimization 
throughout  the  year  would  also  function  more  freely  with  dynamically  simulated  list  construction 
for  the  same  reason. 

Another  strength  of  the  dynamic  simulation  approach  under  the  EER  condition  is  the 
possibility  of  studying  the  effects  of  alternative  DEP  lengths.8  Relatively  long  DEP  periods 
should  allow  the  contract  mission  forecast  feature  of  EPAS  to  produce  better  recruit-MOS 
matches  than  short  periods.  This  analysis  could  not  be  conducted  with  the  fixed  method  of 
generating  REQUEST  lists,  because  the  actual  lists  would  reflect  only  the  current  operational 
DEP  length  policy. 

In  summary,  the  dynamic  simulation  method  of  generating  REQUEST  lists  would 
provide  more  accurate  off-line  simulation  results  than  the  fixed  method  if  the  REQUEST  system 
could  be  adequately  and  accurately  modeled.  This  is  because  the  REQUEST  and  EER 
classification  systems  would  use  the  simulated  recruit  MOS  training  class  choices  and  resulting 
off-line  fill  rates  as  inputs  to  their  PJM  processes  without  being  constrained  by  operational 
assignments  derived  from  the  fixed  actual  REQUEST  lists.  We  describe  above,  in  the  section 
entitled  “Challenges  of  Developing  the  Dynamic  Simulation  Method  of  Generating  the 
REQUEST  List”,  the  complexities  involved  in  accomplishing  this.  We  note  that  there  is  a  large 
subjective  (and  necessary)  component  in  the  Army’s  operational  classification  system  that  would 
be  difficult  to  estimate.  Further,  changes  in  accession  requirements,  recruit  supply,  and  external 


8  This  would  be  the  case  for  the  REQUEST  condition,  too,  but  REQUEST  does  not  include  a  feature  that  uses 
contract  mission  forecasts  to  optimize  person-job  matches  beyond  the  current  month,  as  does  EPAS.  Therefore, 
EPAS  is  more  likely  to  show  practically  significant  benefits  than  REQUEST.  A  test  of  the  two  systems  with 
different  DEP  lengths  would  be  the  ideal. 


economic,  political,  and  social  factors  during  the  FY  would  add  to  the  modeling  process.  In 
addition,  the  REQUEST  modeling  process  would  be  time  consuming  and  expensive. 

Although  the  dynamic  method  has  many  strengths,  we  recommend  that  the  fixed 
approach  be  used  in  the  current  off-line  field  test,  especially  for  the  6-month  analysis  and  the 
monthly  analyses,  if  they  are  included.  We  believe  that  the  expense  of  the  dynamic  method  and 
the  expected  inaccuracies  in  emulating  both  the  systematic  and  subjective  dynamic  factors  in 
REQUEST  would  probably  outweigh  the  error  component  introduced  by  constraining  the 
simulations  in  the  fixed  method  to  use  the  actual  REQUEST  list. 

The  main  strength  of  the  fixed  list  over  the  dynamic  list  approach  discussed  is  the  relative 
ease  of  conducting  the  simulation  procedure  because  the  operational  REQUEST  list  for  each 
recruit  (i.e.,  the  fixed  actual  list)  will  be  contained  in  the  REQUEST  transaction  database.  This 
will  avoid  much  of  the  costs  of  the  dynamic  method.  The  major  weakness  of  the  fixed  list 
method  in  fully  accounting  for  dynamic  changes  in  MOS  and  training  class  fill  patterns,  when 
comparing  REQUEST  and  EER  conditions,  is  that  the  functional  constraint  on  EPAS  will  be 
greater  than  that  on  REQUEST.  The  operational  REQUEST  assignments  will  be  closer  to  the 
simulated  REQUEST  assignments  because  the  PJM  procedures  are  the  same.  The  fixed  list 
approach  will  provide  lower  bound,  or  conservative,  estimates  of  EPAS  benefits,  because  its 
PJM  optimization  algorithm  is  constrained  by  the  fixed  REQUEST  list  generation  method  that 
uses  a  far  less  complicated  list  construction  method.  If  a  more  precise  evaluation  of  EPAS 
potential,  possibly  using  varying  DEP  lengths,  were  desired  now  or  in  the  future,  an 
implementation  of  the  dynamically  simulated  list  construction  approach  would  be  essential. 

In  conclusion,  the  common  dependence  of  the  REQUEST  and  EER  simulated 
classification  systems  on  fixed  actual  REQUEST  MOS  and  training  class  lists  will  not  be  as 
limiting  as  would  initially  appear.  The  fixed  method  is  meaningful  and  defensible  as  the  more 
simplified  off-line  simulation  mechanism  in  light  of  the  currently  short  DEP  lengths,  which  will 
dampen  EPAS  optimization  potential.  EPAS  will  be  able  to  create  PJM  efficiencies  through  its 
optimization  procedure  to  some  extent,  and  in  the  process  accumulate  some  benefits  throughout 
the  FY,  within  the  constraint  of  the  DEP  management  controls  in  effect  during  the  evaluation 
period.  The  fixed  method  of  generating  the  REQUEST  list  will  be  easier  and  less  expensive  to 
develop.  If  a  more  accurate  evaluation  of  EPAS  is  desired  later  on,  then  a  dynamic  simulation 
could  be  considered  with  examination  of  various  DEP  lengths. 

MOS  Assignment  Decision-Making  Model 

The  final  component  in  the  proposed  field  test  is  a  statistical  decision-making  tool  for 
generating  off-line  MOS  training  assignments  from  non-enhanced  and  EER  lists  of  recruits.  It 
may  be  viewed  as  an  analytical  replacement  for  the  actual  assignment  transaction  between  the 
recruit  and  guidance  counselor  at  the  MEPS.  In  other  words,  the  MOS  assignment  decision¬ 
making  model  will  stand  in  for  the  actual  recruit  MOS  training  class  choices  in  the  off-line 
classification  simulations  conducted  in  both  the  REQUEST  and  EER  conditions.  The  decision¬ 
making  model  is  a  necessary  technique  for  conducting  realistic  simulations  of  operational  or 
proposed  classification  systems.  A  brief  description  of  the  model  is  provided  below.  The 
preliminary  technical  overview  is  presented  in  Appendix  A.  A  sample  of  operational  REQUEST 
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transaction  data  is  needed  to  specify  the  model.  This  will  be  conducted  early  on  in  the  field  test 
using  three  months  of  data.  The  model  will  be  validated  and  modified,  if  necessary,  with  the  full 
12-month  data  set. 

The  MOS  assignment  decision-making  model  is  probabilistic.  It  will  statistically 
represent  recruit  MOS  training  class  choices  in  the  off-line  simulations.  The  idea  is  to  specify  an 
equation  that  will  yield  “choice”  probabilities,  which  is  consistent  with  the  recruit  attribute-MOS 
training  class  choice  pattern  revealed  in  the  REQUEST  transaction  data.  In  the  off-line 
simulations,  the  decision-making  model  equation  will  assign  probabilities  to  the  MOS  training 
class  opportunities  that  appear  in  the  REQUEST  or  EER  list  of  a  recruit  in  the  database.  The 
probabilities  will  determine  the  odds  that  an  individual  will  select  each  alternative  MOS  training 
class  start  date  in  the  list,  given  attributes  of  recruits,  the  composition  and  ranking  of  MOS  and 
training  classes,  as  well  as  other  transaction  variables.  Each  recruit’s  synthetic  (i.e.,  simulated) 
MOS  training  class  start  date  choice  will  be  stochastically  selected  in  each  replication  of  the  off¬ 
line  classification  simulations  based  on  the  computed  odds.  This  means  that  the  same  recruit  may 
be  assigned  to  a  different  MOS  training  option  in  his  or  her  list  in  each  replication,  all  else  equal. 

Note  that  each  recruit  in  the  database  actually  enlisted  on  a  specific  contract  date  during 
the  evaluation  period  and  chose  a  specific  MOS  and  training  class.  In  the  REQUEST  and  EER 
simulations  the  recruits’  actual  choices  will  be  ignored,  and  the  recruit  MOS  decision-making 
model  will  be  run  to  make  a  synthetic  MOS  training  class  choice.  Because  the  statistical  model  is 
generated  from  the  REQUEST  transaction  data,  the  results  of  the  simulated  recruit  choices, 
averaged  over  30  replications  of  the  classification  simulations,  are  expected  to  mirror  the 
observable  choice  behavior  of  the  actual  recruits  in  the  operational  Army  classification  process. 

The  task  in  developing  the  stochastic  MOS  assignment  decision-making  model  is  to 
describe  observable  recruit  MOS  training  class  choice  behavior  using  the  variables  available  in 
the  REQUEST  transaction  database.  A  basic  assumption  of  the  model  is  that  recruit  job  choice 
behavior  is  to  some  extent  rational  (i.e.,  not  random).  For  instance,  it  is  not  unreasonable  to 
presume  that  a  recruit  who  is  equally  attracted  to  two  MOS  that  are  presented  at  the  top  of  the 
MOS  training  class  list  would  favor  the  job  with  the  greater  bonus.  Or  suppose  a  recruit  is 
offered  two  MOS  with  aptitude  requirements  that  are  quite  different,  all  other  factors  equal.  The 
guidance  counselor  involved  in  the  transaction  may  point  this  out  to  the  recruit,  influencing  him 
or  her  to  choose  the  job  that  is  closer  to  his  or  her  ASVAB  test  score  profile. 

In  our  modeling  approach,  we  consider  two  types  of  explanatory  variables:  MOS-specific 
and  recruit-specific  attributes.  MOS-specific  variables,  which  may  be  significantly  related  to 
recruit  assignment  choices,  may  include  enlistment  bonuses,  incentives,  term  of  enlistment, 
training  start  dates,  and  MOS  composition  and  rank  order  in  the  list.  An  additional  MOS  attribute 
may  be  its  attractiveness  to  recruits.  Our  preliminary  research  indicates  that  MOS  attractiveness 
variables  (e.g.,  work  requirements  and  ease  of  skill  transfer  to  the  civilian  sector)  may  make 
statistically  significant  contributions  to  the  model.  We  will  only  be  able  to  obtain  a  vague 
indication  from  the  REQUEST  transaction  data  about  whether  this  hypothesis  seems  worth 
pursuing.  We  plan  to  conduct  a  limited  information  gathering  procedure  to  develop  and  measure 
variables  that  tap  this  possible  component  of  the  model.  We  will  do  this  by  conducting  brief 
structured  phone  interviews  with  a  small  sample  of  guidance  counselors  in  a  sample  of  MEPS 
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(Military  Entrance  Processing  Station).  Recruit-specific  attributes  we  will  test  for  inclusion  in  the 
model  will  be  ASVAB  test  scores,  education  level  and  demographics.  Counselor  identity  and 
MEPS  location  will  also  be  considered  as  recruit-specific  explanatory  variables. 
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IV.  FIELD  TEST  DESIGN  DETAILS 


This  chapter  describes  our  technical  approach  in  evaluating  the  impact  of  the  EPAS 
enhancement  of  REQUEST  on  the  Army  recruit  classification  system.  It  will  be  divided  into 
three  sections.  In  the  first  section,  we  give  an  overview  of  the  three  evaluation  periods  under 
which  the  different  field  test  research  questions  will  be  analyzed.  We  also  identify  other  field  test 
parameters  that  will  characterize  the  off-line  simulation  of  the  Army  classification  system,  in 
addition  to  the  REQUEST  and  EER  conditions.  In  the  second  section,  we  provide  the  technics; 
descriptions  of  the  indices  that  will  be  employed  in  analyzing  the  different  study  questions.  The 
third  section  provides  technical  guidelines  in  carrying  out  the  different  analyses  under  each  of  the 
three  evaluation  periods. 


Overview 


Three  Sets  of  Evaluation  Periods 

Initial  monthly  assessments  of  the  impacts  of  EPAS  enhancement  of  REQUEST  will  be 
carried  out  using  REQUEST  transaction  data  from  the  first  six  months  of  the  field  test  period. 
These  monthly  analyses  will  be  limited  to  two  or  three  of  the  first  six  months  of  the  field  test. 
The  main  advantage  of  these  preliminary  analyses  is  that  they  would  provide  timely  results 
regarding  the  classification  benefits  of  EPAS  enhancements  of  REQUEST.  These  results, 
however,  are  not  cumulative  and  therefore  are  expected  to  underestimate  potential  EPAS 
benefits.  The  preliminary  monthly  analysis  will  be  based  on  the  fixed  actual  REQUEST  list. 

A  second  set  of  intermediate  analyses  will  be  carried  out  using  a  six-month  evaluation 
period,  again  based  on  the  fixed  actual  REQUEST  list.  Results  from  these  intermediate  analyses 
could  begin  to  show  cumulative  effects  of  EPAS,  even  if  they  will  remain  constrained  by  fixed 
actual  REQUEST  training  choices  of  recruits.  These  results  are  expected  to  provide  better 
estimates  of  the  impact  of  EPAS  enhancements  on  REQUEST  compared  to  the  results  of  the 
preliminary  analyses. 

A  final  set  of  analyses  will  be  carried  out  for  the  full  fiscal  year  evaluation  period.  These 
analyses  will  be  carried  out  using  fixed  actual  REQUEST  lists  or  dynamically  generated 
REQUEST  lists,  depending  on  the  feasibility  of  the  latter.  The  results  from  these  12-month  period 
analyses  will  represent  the  variety  of  recruits  and  conditions  met  during  a  fiscal  year. 
Consequently,  these  results  will  more  accurately  reflect  the  overall  cumulative  impact  of  EPAS  on 
REQUEST  compared  to  the  preliminary  monthly  and  intermediate  analyses,  regardless  of  the  type 
of  the  list  that  will  be  employed.  A  more  accurate  measure  of  potential  EPAS  cumulative 
classification  benefits  is  expected  under  dynamically  generated  REQUEST  lists,  if  feasible, 
compared  to  the  estimates  that  will  be  obtained  under  fixed  actual  REQUEST  lists. 

Additional  Classification  Condition 

Underlying  the  simulation  in  the  three  evaluation  periods  is  a  type  of  job  composite, 
which  will  impact  the  analyses  for  all  three  types  of  research  questions.  The  job  composite  of  a 
classification  system  plays  a  dual  role  in  the  overall  classification  process.  First,  composites  are 
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employed  in  the  initial  construction  of  the  list  through  cut  scores.  Only  MOS  for  which  the 
recruit  meets  the  minimum  standard  in  the  respective  job  composite  can  be  considered  in  the  list 
of  this  recruit.  Second,  composites  also  provide  guidance  in  matching  the  individual  to  jobs  in 
which  he  or  she  is  best  suited.  Informally,  this  is  part  of  REQUEST  in  that  job  composites  are 
available  to  the  career  counselor  for  purposes  of  advising  on  a  recruit’s  MOS  choice  decision.  In 
an  EER  system,  this  person-job  matching  guidance  is  formally  factored  in  the  reordering  of  MOS 
in  the  EER  list. 

Given  the  important  role  of  job  composites,  the  manner  by  which  these  are  computed  is 
central  in  the  overall  performance  of  a  classification  system.  In  this  study,  the  impact  of  EPAS 
enhancement  on  REQUEST  will  be  evaluated  under  two  sets  of  classification  composites.  The 
first  set  of  composites  will  employ  the  interim  least-square  weights  of  seven  ASVAB  subtest 
scores  on  the  existing  nine  Army  aptitude  areas.  The  second  set  of  composites  will  be  based  on 
the  least-square  weights  of  seven  ASVAB  subtest  scores  on  the  17  job  family  configuration  of 
Johnson  and  Zeidner.9 

An  unresolved  problem  related  to  job  classification  composites  is  how  to  handle 
eligibility  of  recruits  to  various  MOS,  if  the  job  composite  parameter  is  set  to  something  other 
than  the  current  nine  family  unit  weighted  composites.  For  instance,  “equivalent”  cut  scores  will 
need  to  be  defined  before  off-line  evaluations  can  begin.  This  will  not  be  a  problem  if  the  job 
composite  parameter  is  the  same  as  the  set  of  composites  that  are  operational  during  the  field 
test,  as  recruits  will  be  eligible  for  all  jobs  in  their  actual  REQUEST  lists. 

Simulation  Condition  Combinations 

Employing  all  combinations  of  conditions  and  parameters  in  the  field  test  will  entail  a 
large  analytical  and  computing  effort.  The  combinations  of  evaluation  periods,  job  composites, 
and  type  of  classification  system  are  shown  in  the  table  below.  Using  only  three  preliminary 
monthly  analyses,  this  table  represents  a  total  of  20  separate  simulations,  each  of  which  will 
involve  a  total  of  30  replications.  Separate  job  list,  classification  efficiency  and  accession 
analyses  will  be  carried  out  for  each  of  the  10  corresponding  pairs  of  REQUEST  and  EER 
simulations. 


Table  1 

Potential  Field  Test  Classification  Conditions  and  Job  Family  Composites  by  Potential 
Evaluation  Periods 


Evaluation  Period 

Classification  System 

Total 

9-JF 

17-JF 

REQUEST 

EPAS-enhanced 

REQUEST 

(3)  Monthly  Analyses 

3 

3 

3 

3 

12 

6-Month  Analyses 

1 

1 

1 

1 

4 

12-Month  Analyses 

1 

1 

1 

1 

4 

9  For  a  description  of  the  17  job  family  configuration,  see  P.  Greenston  (2002),  Proposed  New  Army  Aptitude  Area 
Composites:  A  Sumary  of  Research  Results,  Study  Report  2002-03.  Alexandria,  VA:  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences. 
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We  recommend  that  the  preliminary  and  intermediate  evaluations  focus  on  the 
combination  of  job  composites  and  optimization  that  more  likely  will  be  recommended  for 
implementation,  so  as  not  to  defeat  the  advantage  of  timely  feedback  from  these  initial 
evaluations.  For  the  final  evaluation,  a  judicious  choice  of  study  parameter  combinations  can  be 
considered. 


Analysis  Indices 

Detailed  descriptions  of  the  field  test  analysis  indices  are  organized  into  three  sections 
according  to  the  type  of  research  question  under  consideration,  without  regard  to  the  length  of 
evaluation  period.  In  order,  these  are  job  list  analysis  indices,  classification  efficiency  analysis 
indices,  and  accession  analysis  indices. 

Job  List  Analysis 

The  indices  that  will  be  employed  in  the  job  list  comparison  analyses  are  described 
below.  The  discussion  is  structured  by  specific  job  list  comparison  research  question  associated 
with  the  indices. 

Size  of  Intersection  Between  REQUEST  and  EPAS  Lists  Analysis 

The  first  type  of  job  list  analysis  will  examine  the  size  of  intersection  between  the 
REQUEST  list  and  EPAS  list  of  job  opportunities.  In  this  analysis,  the  length  of  the  intersection 
for  each  recruit  will  be  represented  either  by  the  count 

^number  of  common  MOS  -  class  start  dates  in  the  REQUEST  and 
VEPAS  lists  of  the  ith  recruit  at  the  rth  assignment  replication 


YN(i,r)  = 


or  the  equivalent  form  in  proportion 

^number  of  common  MOS  -  class  start  dates  in  the  REQUEST  and 
,  ,  ^EPAS  lists  of  the  z'th  recruit  at  the  rth  assignment  replication  y 

r  length  of  REQUEST  list  of  the  zth  recruit 

These  two  forms  of  the  intersection  will  be  employed  in  the  computation  of  the  following  job  list 
analysis  indices. 

Overall  Analysis.  The  intersection  between  REQUEST  and  EPAS  lists  of  all  recruits  in 
the  simulation  of  a  specified  evaluation  period  will  be  summarized  by  replicate  using  the  indices 
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Mr)  =  >W(l,'-)+yM2,r)+yiV(3.r)-t-- 

total  recruits  in  the  evaluation  period 


Jp(r)  =  yp(ur)+yp(2,r)+y/^,r)+- 

total  recruits  in  the  evaluation  period 

The  index  YN(r)  is  just  the  average  of  the  number  of  common  MOS-class  start  dates  of  all 

recruits  in  the  rth  simulation  replication.  The  index  YP(r)  is  an  analogous  average  but  calculated 
using  the  proportion  form  of  the  size  of  intersection  relative  to  the  length  of  the  respective 
REQUEST  lists  of  recruits. 

The  proportion  form  of  the  index  is  recommended  over  the  count  form  if  the  lengths  of 
REQUEST  lists  vary  substantially  across  recruits  for  the  following  reason.  In  the  computation  of 

YN(r),  recruits  with  longer  REQUEST  lists  likely  will  be  weighted  more  than  recruits  with 

shorter  lists.  This,  however,  will  not  be  a  concern  for  YP(r)  as  the  contributions  of  all  recruits 
are  put  in  the  same  scale,  without  regard  to  the  actual  lengths  of  their  respective  REQUEST  lists. 

Subgroup  Analysis.  A  separate  set  of  indices  will  be  computed  using  the  same  simulation 
replicate  for  conducting  subgroup  analysis  of  the  size  of  intersection  between  REQUEST  and 
EPAS  lists,  based  on  recruit  gender,  AFQT,  and  education  levels.  The  names  used  to  label  these 
three  subgroup  factors  are  shown  in  the  table  below  under  the  CAT  column.  Also  shown  in  the 
table  are  the  subgroup  levels  that  will  be  analyzed  for  each  factor,  which  are  shown  inside  the 
parenthesis  under  the  GRP  column  next  to  the  their  respective  group  value.  The  variables  CAT 
and  GRP  are  used  in  the  subgroup  indices  below  and  in  the  remainder  of  this  section,  to  represent 
in  general  notation  the  factor  (CAT)  and  a  specific  subgroup  (GRP)  combination. 

Table  2 


Subgroup  Analysis  Factors  and  Levels 


Type  of  Subgroup  Analysis 

CAT 

GRP 

By  recruit  gender 

SEX 

M  (male),  F  (female) 

By  recruit  AFQT 

TSC 

A  (I-mA),  B  (DIB),  C  (IV) 

By  recruit  education  level 

EDUC 

G  (HS  Grad.),  S  (Senior),  N  (Non-grad.) 

In  general,  to  carry  out  subgroup  analysis  of  size  of  intersection  using  factor  CAT,  the 
intersection  between  the  REQUEST  and  EPAS  lists  will  be  summarized  by  subgroup  level  of  the 
factor  for  each  simulation  replication.  The  count  and  proportion  forms  of  the  index  for  a  specific 
factor  by  subgroup  combination  are 
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YN  cat  (GRP,  r)  = 


sum  of  YN(i,  r)for  all  recruits  in  subgroup  GRP  of  factor  CAT 
total  recruits  in  subgroup  GRP  of  factor  CAT 


YPcat  {GRP,  r)  = 


sum  of  YP(i,  r) for  all  recruits  in  subgroup  GRP  of  factor  CAT 
total  recruits  in  subgroup  GRP  of  factor  CAT 


These  indices  are  just  the  averages  of  the  count  (YN)  or  proportion  (YP)  of  common  MOS-class 
start  dates  of  recruits  in  the  rth  simulation  replication  computed  by  subgroups  of  CAT. 

We  briefly  illustrate  the  subgroup  notation  above  by  looking  at  the  subgroup  analysis  by 
gender.  For  this  example,  the  factor  CAT  equals  SEX,  and  the  relevant  subgroup  values  are 

GRP=M,F.  Using  the  count  form  of  the  index,  we  will  compute  YN  sex  ( M,r )  and  YN  sex  ( F,r ) 
to  represent  the  average  number  of  common  MOS-class  start  dates  for  male  and  female  recruits, 
respectively,  in  the  rth  simulation  replication.  The  analysis  of  interest  in  this  example  would  be 
the  comparison  of  the  size  of  intersection  of  REQUEST  and  EPAS  lists  between  male  and 
female  recruits. 

Quality  of  the  Intersection  Between  REQUEST  and  EPAS  Lists  Analysis 

The  preceding  indices  of  size  of  intersection  are  based  on  YN(i,r),  the  simple  count  of 
common  MOS-class  start  dates  in  the  REQUEST  and  EER  lists  of  the  ith  recruit.  A  limitation  of 
the  measures  of  size  of  intersection  based  on  YN(i,  r)  is  that  all  common  MOS-class  start  dates 
contribute  equally,  regardless  of  their  respective  rank  orders  in  the  EPAS  list,  or  their  respective 
movements  in  rank  order  locations  from  the  original  REQUEST  list  to  the  EER  list. 

The  EPAS  rank  order  and  movement  in  REQUEST  rank  order  locations  are  two 
important  criteria  in  constructing  an  index  of  the  “quality”  of  the  intersection.  In  addition  to 
counting  the  number  of  common  MOS-class  start  dates,  this  index  also  provides  an  indicator  of 
their  contributions  to  the  overall  EPAS  person  job-matching  enhancement  of  the  REQUEST  list. 
Using  these  two  criteria,  for  .example,  we  would  give  more  weight  to  an  MOS-class  start  date 
that  moves  from  the  bottom  of  the  original  REQUEST  list  to  near  the  top  of  the  EER  list. 
Moreover,  its  contribution  to  the  value  of  the  quality  index  of  intersection  will  be  larger  if  it  is  at 
the  same  time  close  to  the  top  of  the  EPAS  list. 

The  indices  that  will  follow  are  based  on  QP{i,r),  which  represents  the  quality  of 
intersection  between  the  REQUEST  and  EPAS  lists  of  the  ith  recruit.  This  quality  value  is  of  the 
form 

Opi.  x  =  gM(l,i,r)+eM(2,/,r)+i2M(3,i,r)+-  ^ 

length  of  the  REQUEST  list  of  the  ith  recruit  ’ 

which  is  the  average  of  quality  contributions  of  the  MOS  training  opportunities  in  the  REQUEST 
list  of  the  ith  recruit.  The  numerator  above  is  the  sum  of  the  “QM"  values  of  the  MOS  training 
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opportunities  in  the  REQUEST  list  of  the  /th  recruit.  The  exact  definition  of  QM(m,i,r)  is  given 
in  Appendix  B. 

Less  formally,  QM(m,i,  r )  is  equal  to  0  if  the  mth  REQUEST  training  opportunity  of  the 
ith  recruit  is  not  in  the  EPAS  list.  QM(m,i,r)  is  equal  to  1  if  the  mth  REQUEST  training 
opportunity  is  in  the  EPAS  list,  but  its  priority  rank  order  in  the  EP AS-enhanced  REQUEST  list 
is  lower  than  originally  in  REQUEST.  QM(m,i,r )  is  a  number  between  1  and  2  if  the  mth 
REQUEST  training  opportunity  is  in  the  EPAS  list,  and  its  priority  rank  order  in  the  EP  AS- 
enhanced  REQUEST  list  is  higher  than  originally  in  REQUEST.  (The  actual  value  of 
QM(m,i,r)  is  closer  to  2  than  to  1  if  the  mth  REQUEST  training  opportunity  appears  near  the 
top  of  the  EPAS  list  and,  originally,  near  the  bottom  of  the  REQUEST  list.)  A  more  rigorous 
discussion  of  QM{m,i,r)  is  given  in  Appendix  B. 

Overall  Analysis.  The  quality  of  the  intersection  between  the  REQUEST  and  EER  lists 
of  recruits  will  be  summarized  by  simulation  replicate  using  the  index 

gp(r) .  QP(U)+QP(2,r)+QP(\r)+- 
total  recruits  in  the  evaluation  period 


This  is  just  the  average  of  the  quality  of  intersections  across  all  recruits  in  the  evaluation  period. 
From  the  description  of  QP(i,r )  above,  it  is  not  difficult  to  verify  that  YP(r )  <  QP(r)  <  2YP(r) . 

Given  this  relationship,  the  difference  QP(r)~  YP(r)  would  be  a  meaningful  index  of  potential 
EPAS  enhancement  that  is  due  solely  on  its  reordering  of  the  original  REQUEST  list,  without 
regard  to  the  number  of  matching  MOS-class  start  dates  in  the  intersection. 

Subgroup  Analysis.  The  indices  that  will  be  employed  in  the  subgroup  analysis  of  quality 
of  intersection  between  the  REQUEST  and  EPAS  lists  of  recruits  will  be  constructed  as  before. 
To  carry  out  a  subgroup  analyses  based  on  a  fixed  factor  CAT,  the  index  of  the  quality  of 
intersection 


QPaT(GRP,r)  = 


sum  of  QP(i,  r) over  all  recruits  in  subgroup  GRP  of  factor  CAT 
total  recruits  from  subgroup  GRP  in  the  evaluation  period 


will  be  computed  for  all  possible  subgroup  values  of  GRP  associated  to  the  factor. 


Size  of  Intersection  Disregarding  Date  of  Availability 


A  slightly  modified  length  of  intersection  may  also  be  considered  by  ignoring  class  start 
dates,  and  instead  using  only  the  MOS  to  match  training  opportunities  in  the  REQUEST  and 
EPAS  lists  of  recruits.  Using  this  approach,  a  training  opportunity  in  the  REQUEST  list  will  be 
counted  if  the  associated  MOS  appears  in  the  EPAS  list,  even  if  the  month  of  the  REQUEST 
class  start  date  does  not  match  any  EPAS  class  month  with  the  same  MOS. 


The  modified  length  of  intersection  for  each  recruit  will  be  denoted  in  count  form  by 
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YN*(i,r) 


^  number  of  training  opportunities  in  the  REQUEST  list  oU 
the  ith  recruit  with  matching  MOS  in  the  EPAS  list  j 


or  in  proportion  form  by 


f  number  of  training  opportunities  in  the  REQUEST  list  of 

x  the  ith  recruit  with  matching  MOS  in  the  EPAS  list 

YP  (i  r)  =  - - - - 

v  ’  ’  length  of  REQUEST  list  of  the  ith  recruit 

Job  list  comparison  analyses  based  on  YN*(i,  r)  or  YP*(i,r)  can  be  useful  as  follow  up  to 
analyses  based  on  the  first  length  of  intersection,  for  examining  whether  EPAS  is  capturing  the 
MOS  but  missing  out  on  class  months.  In  turn,  this  information  can  be  used  to  recommend 
possible  directions  as  to  how  EPAS  constructs  the  optimal  guidance  list  or  can  suggest 
alternative  REQUEST  and  EPAS  list  merge  rules. 

Overall  Analysis.  The  length  of  intersection  between  REQUEST  and  EPAS  list  based 
solely  on  matching  MOS  will  be  summarized  for  all  recruits  in  the  evaluation  period  by 
simulation  replicate,  using  the  indices 


— • ,  ,  =  HV(l,r)+W(2,r)-fyW(3.r)+- 
total  recruits  in  the  evaluation  period 


yf  !r\_  YP-(\,r)+YP-{2,r)+YP’(Xr)+- 
total  recruits  in  the  evaluation  period 


Again,  these  are  simple  averages,  respectively,  of  the  count  and  proportion  forms  of  the  modified 
length  of  intersection  computed  using  all  recruits  in  the  evaluation  period. 

Subgroup  Analysis.  The  subgroup  analysis  of  the  modified  length  of  intersection  of  a 
specified  factor  CAT  will  be  based  on  the  indices 


YNcAr(GRP,r) 


sum  of  YN*  (i,  r)  over  all  recruits  in  subgroup  GRP  of  factor  CAT 
total  recruits  from  subgroup  GPP  in  the  evaluation  period 


YPcat  (GRP,  r) 


sum  of  YP*  (i,  r)  over  all  recruits  in  subgroup  GRP  of  factor  CAT 
total  recruits  from  subgroup  GRP  in  the  evaluation  period 


that  will  be  computed  for  all  levels  or  subgroup  values  GRP  of  the  factor. 
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Analysis  of  Priority  MOS 

The  analysis  of  priority  MOS  will  examine  the  impact  of  EPAS  enhancement  on  the 
MOS  choices  of  recruits  by  comparing  the  number  of  Priority-25  MOS  at  the  top-n  of  the 
REQUEST  and  EER  lists.  This  analysis  will  indicate  the  consistency  of  the  EPAS  enhancement 
relative  to  Army  priorities,  as  reflected  by  the  number  of  Priority-25  MOS  in  the  reordered 
REQUEST  list.  The  value  of  n  will  be  fixed  in  this  analysis. 

This  analysis  will  compare  the  ordered  MOS  training  opportunities  in  the  REQUEST  list 
of  a  recruit  under  the  REQUEST  simulation  condition  to  the  MOS  training  opportunities  in  the 
EER  list  of  the  same  recruit  under  the  EER  simulation  condition.  (These  are  the  MOS  training 
opportunities  that  are  visible  to  the  recruit  under  the  two  classifications  systems.)  Analysis 
indices  will  be  constructed  separately  under  the  two  classification  system  conditions  based  on  the 
quantities 


YN^  (i,  r )  =  count  of  Priority  -  25  MOS  at  the  top  -  n  of 
(assignment)  REQUEST  list  of  the  ith  recruit 
YN^\i,  r)  =  count  of  Priority  -  25  MOS  at  the  top  -  n  of 

EPAS  -  enhanced  REQUEST  list  of  the  ith  recruit. 

which  are  respectively  derived  from  the  assignment  REQUEST  list  and  EP AS-enhanced 
REQUEST  list. 

Overall  Analysis.  The  number  of  priority  MOS  at  the  top-n  of  REQUEST  and  EPAS- 
enhanced  REQUEST  lists  will  be  summarized  by  simulation  replicate  using  the  indices 

_  yiVM(l.r)+Wl-)(2,r)+y4-)(3.r)+- 
total  recruits  in  the  evaluation  period 

YNi?(r)=  ^YM+YN^rj+rNY^r)*- 
total  recruits  in  the  evaluation  period 

These  indices  are  simple  averages  of  the  number  of  top-n  priority  MOS  of  all  recruits  in  an 
evaluation  period.  As  noted  later  on,  the  intra-individual  differences  YN^(i,r)-YN^(i,r)  will 
play  a  role  in  evaluating  the  differences  between  these  two  indices,  as  well  as  between  the 
subgroup  indices  below. 

Subgroup  Analysis.  The  subgroup  analysis  comparing  priority  MOS  at  the  top-n  of 
REQUEST  and  EP  AS-enhanced  REQUEST  lists  will  be  based  on 
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— («)  /  x  sum  of  YN$\i,  r)over  all  recruits  in  subgroup  GRP  of  factor  CAT 

YN  r,cat  (GRP,  r)  = - - - ; — ; - - - — — — : - ; — : - : — : - 

total  recruits  m  subgroup  GRP  in  the  evaluation  period 

—An)  /  \  sum  of  YN(f\i,  r)over  all  recruits  in  subgroup  GRP  of  factor  CAT 

YN e.cat  (GRP,  r)  = - - - : — ; - - - ■_  ■  ■  . — ; - : — : - : — ; - 

total  recruits  in  subgroup  GRP  in  the  evaluation  period 

These  averages  will  be  computed  for  all  subgroup  levels  GRP  of  the  factor  CAT: 

Classification  Efficiency  Analysis 

The  classification  efficiency  analysis  will  examine  how  EPAS  enhancement  impacts  the 
person-job  matching  efficiency  of  the  REQUEST  system.  The  mean  predicted  performance 
(MPP)  of  recruits  based  on  their  MOS  assignments  under  REQUEST  and  EER  classification 
systems  will  be  compared.10  The  analysis  will  be  carried  out  at  the  Army  organization-wide  level 
and  also  at  the  MOS  level  using  the  indices  described  below.  The  MOS  level  analysis  will  be 
useful  in  identifying  MOS  that  are  negatively  impacted  in  terms  of  recruit  predicted 
performance,  if  any,  by  EPAS  enhancement  of  REQUEST. 


Army  Organization-wide  Level  Classification  Efficiency 

The  predicted  performance  of  the  ith  recruit  is  a  weighted  sum  of  the  recruit’s  ASVAB 
subtest  scores,  with  weights  that  will  depend  on  the  MOS  job  assignment  of  the  recruit.  It  will 
represent  his  or  her  contribution  in  the  classification  efficiency  indices.  The  predicted 
performance  of  the  ith  recruit  will  be  denoted  separately  under  REQUEST  and  EER  simulation 
conditions  by  PP(i,hR(i,r ))  and  PP(i,hE(i,r)),  respectively.  In  these  expressions,  hR(i,r)  and 
hE(i,r )  respectively  indicate  the  MOS  that  is  assigned  to  the  ith  recruit  in  the  rth  classification 
simulation  replicate  through  the  MOS  assignment  decision  model. 

The  classification  efficiency  of  the  Army  organization-wide  MOS  assignments  of  recruits 
in  an  evaluation  period  will  be  summarized  by  simulation  replicate  using  the  indices 


10  Different  sets  of  classification  composites  will  be  used  to  make  assignments  in  the  field  test.  Therefore,  a  single 
set  of  classification  composite  weights,  which  represents  relevant  measures  of  performance  in  the  MOS  in  all 
conditions,  is  needed  to  compute  equivalent  classification  efficiency  indices  across  the  conditions.  These  composites 
are  referred  to  as  evaluation  composites.  The  weights  in  the  composite  associated  with  the  MOS  to  which  a  recruit 
was  assigned  are  applied  to  the  ASVAB  test  scores  of  the  assigned  recruit.  If  the  classification  and  evaluation 
composite  weights  are  computed  from  the  same  sample,  then  a  cross-validation  procedure  must  be  used  to  create 
independent  sets  of  weights.  A  recruit’s  composite  score  in  the  job  to  which  he  or  she  is  assigned,  computed  from 
the  evaluation  composites,  is  called  the  predicted  performance  score.  The  average  of  these  scores  across  all  recruits 
and  MOS  in  a  single  classification  replicate  is  mean  predicted  performance  (MPP),  the  index  of  classification 
efficiency  for  the  replicate.  The  MPP  index  for  an  entire  simulation  is  the  average  MPP  across  replicates. 


MPP  (, )  PP(l.ft»(i,r))+PP(2,V(2,r))+fP(3>J,(3,r))+- 

R  total  recruits  in  the  evaluation  period 


MPP  (j)  PPMlr)hPP(2M2,r)hPP(3.hA3.r)h-  - 

E  total  recruits  in  the  evaluation  period 

respectively,  under  REQUEST  and  EER  simulation  conditions.  These  indices  are  the  simple 
averages  or  means  of  predicted  performances  under  the  two  classification  systems.  Each  index 
can  vary  across  simulation  replications  depending  on  the  MOS  assigned  to  recruits  by  the  MOS 
assignment  decision  model  in  the  independent  replications. 

MOS  Level  Classification  Efficiency 

The  classification  efficiency  of  MOS  assignments  of  recruits  at  the  MOS  level  will  be 
summarized  separately  under  the  two  classification  conditions  by  replicate  using  the  indices 

/  x  sum  of  PP(i,  hK  (i,  r))  of  recruits  in  the  mth  MOS  during  the  rth  replicate 

J\*fPP  (/ft  r)~ - - — — — — - 

f  total  recruits  in  the  mth  MOS  during  the  rth  replicate 


,  /  \  sum  of  PP(i,  hr  (/, r))  of  recruits  in  the  mth  MOS  during  the  rth  replicate 

JyjPP  1/72  T  )  — - - - =— - — — - 

£  ’  total  recruits  in  the  mth  MOS  during  the  rth  replicate 

where  m  indicates  a  specific  MOS.  These  indices  are  based  on  the  same  recruit  MOS 
assignments  used  in  the  Army  organization-wide  level  indices,  but  employing  only  recruits 
assigned  to  the  mth  MOS.  The  comparison  of  MPPR(m,r )  and  MPPE(m,r)  will  look  at  possible 
differences  in  classification  efficiency  at  the  MOS  level  under  the  two  conditions. 

Accession  Analysis 

The  third  type  of  analysis  will  focus  on  how  EPAS  enhancement  of  REQUEST  impacts 
Army  accession  goals.  Accession  analyses  that  are  considered  in  this  study  are  divided  into  three 
parts.  First,  we  look  at  the  overall  Army  monthly  accession  goals  under  REQUEST  and  EPAS- 
enhanced  REQUEST  assignment  conditions.  Second,  we  examine  monthly  accession  goals  of 
priority  MOS  under  the  two  conditions.  The  third  set  of  analyses  are  again  concerned  with 
priority  MOS  but  focusing  on  fiscal  year  accession  goals.  The  relevant  evaluation  indices  in  each 
of  these  analyses  are  described  separately  below. 

Overall  Army  Monthly  Accession 

The  first  accession  analysis  will  provide  information  on  how  assignments  based  on 
EPAS-enhanced  REQUEST  will  compare  to  REQUEST  assignments  in  meeting  overall  monthly 
Army  accession  goals.  The  month  unit  of  time  in  this  analysis  refers  to  the  receiving  station 
month  (RSM),  not  the  contract  month,  of  a  recruit. 
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The  overall  Army  monthly  accessions  will  be  summarized  separately  under  REQUEST 
and  EER  classification  conditions  by  simulation  replicate  using  the  indices 


AP,(l,r)= 


overall  Army  total  accession  for  the  /th  RSM  during  the 
rth  replication  of  REQUEST  assignments 
overall  Army  target  accession  for  the  /th  RSM 


APe(l,r)  = 


overall  Army  total  accession  for  the  /th  RSM  during  the 
rth  replication  of  EPAS  enhanced  REQUEST  assignments 
overall  Army  target  accession  for  the  /th  RSM 


These  indices  are  just  the  proportion  of  overall  Army  target  accession  met  for  the  /th  RSM 
during  the  rth  replication 

Priority  MOS  Monthly  Accession 

The  second  set  of  accession  analyses  will  examine  how  EP AS-enhanced  REQUEST  and 
REQUEST  recruit  assignments  compare  in  meeting  monthly  accession  goals  of  priority  MOS. 
The  monthly  accessions  of  priority  MOS  will  be  summarized  by  replicate  using  the  indices 


APf(l,m,r)- 


mth  priority  MOS  total  accession  for  the  /th  RSM  during  the 
rth  replication  of  REQUEST  assignments 
mth  priority  MOS  target  accession  for  the  /th  RSM 


APg(l,m,  r)  = 


mth  priority  MOS  total  accession  for  the  /th  RSM  during  the 
rth  replication  of  EPAS  enhanced  REQUEST  assignments 
mth  priority  MOS  target  accession  for  the  /th  RSM 


The  indices  AP/  (l,m,  r)  and  AP/  (l,m,  r)  represent  the  proportion  of  the  mth  priority  MOS 
target  accession  met  for  the  /th  RSM  during  the  rth  replication,  respectively,  under  REQUEST 
and  EER  classification  conditions. 


Priority  MOS  Fiscal  Year  Accession 

The  last  set  of  accession  analyses  will  look  at  how  EPAS -enhancement  impacts  fiscal 
year  accession  goals  of  priority  MOS.  This  is  essentially  a  repetition  of  the  preceding  analysis, 
but  applied  to  the  full  fiscal  year  accession  goals  of  the  mth  priority  MOS. 

The  fiscal  year  accessions  of  priority  MOS  will  be  summarized  by  replicate,  respectively, 
under  REQUEST  and  EER  conditions  using  the  indices 
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APRPY(m,r) 


Ymth  priority  MOS  total  accession  for  the  fiscal  year  during  the N 
v  rth  replication  of  REQUEST  assignments  y 

mth  priority  MOS  target  accession  for  the  fiscal  year 


^rath  priority  MOS  total  accession  for  the  fiscal  year  during  the N 

,  v  1  rth  replication  of  EPAS  enhanced  REQUEST  assignments  , 

AP  [fti  v )  ""  —  '  — ■'  ■■  -  — - — — . . 

£  v  ’  '  mth  priority  MOS  target  accession  for  the  fiscal  year 

These  indices  are  the  proportion  of  the  mth  priority  MOS  target  accession  met  for  the  fiscal  year 
during  the  rth  replication. 


Analyses  By  Evaluation  Periods 

The  following  discussion  describes  the  analyses  that  will  be  carried  out  at  different 
evaluation  periods.  It  provides  an  overview  of  the  simulation  experiment  at  each  evaluation 
period,  the  questions  that  will  be  considered,  and  a  summary  of  evaluation  indices  that  are 
relevant  for  the  different  types  of  analyses.  Technical  issues  that  are  considered  to  be  important 
in  the  proper  implementation  of  the  analyses  are  mentioned  in  the  discussion. 


Preliminary  Monthly  Analyses 

We  will  conduct  preliminary  analyses  using  recruit  cohorts  from  each  of  two  or  three 
months  selected  from  the  first  six  contract  months  in  the  field  test.  Analysis  results  will  be 
reported  separately  for  each  evaluation  month  as  REQUEST  transaction  data  become  available. 
Only  research  questions  related  to  job  list  comparisons  and  classification  efficiency  benefits  will 
be  considered  in  these  preliminary  analyses,  which  will  be  based  on  fixed  actual  REQUEST  lists. 
Given  the  monthly  nature  of  the  evaluation  period,  examining  accession  goals  at  this  stage  is  not 
very  meaningful  as  recruits  who  report  during  a  given  receiving  station  month  ordinarily  come 
from  more  than  one  contract  month. 

Job  List  Analyses 

Alternative  descriptive  analyses  comparing  REQUEST  and  EPAS  lists  will  be  carried  out 
using  recruit  cohort  from  each  of  the  selected  months.  These  analyses  are  grouped  into  two  parts 
in  the  following  discussion.  The  first  part  deals  with  the  different  ways  of  evaluating  the  size  of 
intersection  between  the  two  lists,  while  the  second  part  pertains  to  the  analysis  of  top-n  priority 
MOS. 


A  brief  overview  of  the  analysis  using  the  length  of  intersection  index  is  as  follows.  The 
discussion  is  based  on  the  YN  index,  but  will  also  apply  to  the  YP  and  QP  indices.  The  three 
indices  corresponding  to  different  representations  of  the  size  of  intersection  are  shown  in  Table 
3,  along  with  “replicate  observations”  from  which  they  are  computed.  These  expressions  were 
described  earlier  in  detail. 
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Table  3 

Job  List  Analysis  of  the  Length  of  Intersection 


Analysis  Index 

Replicate 

Observations 

Subgroup  Analysis 
Factor 

Length  of  Intersection 

YN{r) 

YN(i,r ) 

(Overall) 

YN  cat  (GRP,  r) 

YN{i,r) 

SEX,  TSC,  EDUC 

Proportion  of  Intersection 

YP{r) 

YP(i,r ) 

(Overall) 

YPCAr(GRP,r ) 

YP(i,r ) 

SEX,  TSC,  EDUC 

Quality  of  Intersection 

QP(r) 

QP(i,r) 

(Overall) 

QP  CAT  iffRPi  f) 

QP(i.r) 

SEX,  TSC,  EDUC 

Observations  YN(i,r)  will  be  computed  for  the  zth  recruit  at  the  rth  simulation  replicate 
of  a  given  month.  The  sample  size  of  YN  observations  will  be  equal  to  the  number  of  contracts 
during  the  month.  (This  is  naturally  fixed  over  assignment  replications  r=l,2,...,30.)  Each 
YN (z,  r)  will  represent  the  number  of  matching  MOS  between  the  recruit’s  fixed  actual 
REQUEST  and  EPAS  lists  at  the  rth  replication. 

The  sample  YN  observations  will  be  analyzed  using  numerical  or  graphical  descriptive 
statistics.  In  addition  to  the  index  YN(r)  this  analysis  can  also  include  percentiles  to  summarize 
the  YN  (z,  r )  of  all  recruits  in  the  month.  Computation  of  these  descriptive  statistics  will  be 
carried  out  over  all  30  independent  replications.  This  analysis  will  provide  information  about  the 
typical  length  of  intersection  between  EPAS  and  REQUEST  lists,  along  with  an  associated 
measure  of  sampling  variation,  over  monthly  recruit  assignments. 


Descriptive  analysis  of  the  length  of  intersection  between  REQUEST  and  EPAS  lists  will 
also  be  carried  out  by  recruit  gender,  AFQT  category,  and  education  level.  Statistics 
summarizing  the  intersection  between  the  two  lists  will  be  compared  across  levels  of  each  of 
these  factors.  These  analyses  will  be  helpful  in  detecting  potential  differential  impact  of  EPAS 
enhancement  on  MOS  training  opportunities  of  recruits. 


The  second  set  of  descriptive  analyses  will  compare  the  number  of  priority  MOS 
included  at  the  top-n  of  REQUEST  and  EER  lists.  The  relevant  indices  and  observations  are 
shown  in  Table  4.  Unlike  in  the  job  lists  intersection  analyses,  for  the  zth  recruit,  separate 
“observations”  will  be  constructed  from  each  type  of  list.  The  matched  pairs  of  observations  are 

given  by  YNg\i,r)  and  KVc^z'.r).  The  analysis  here  will  focus  on  the  intra-recruit  difference 


.  This  difference  is  meaningful  as  the  correspondence  between  recruit 
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identity  and  contract  date  in  the  database  will  be  retained  in  the  off-line  REQUEST  and  EER 
simulated  assignments.  The  overall  structure  of  the  analysis  will  be  the  same  as  in  analyses 
pertaining  to  job  list  intersection. 


Table  4 

Job  List  Comparison  Analysis  of  Count  of  Top-n  Priority  MOS 


Analysis  Index 

Replicate 

Observations 

Main  Analysis  Factor 
(Type  of  List) 

Subgroup  Analysis 
Factor 

YN{R](r),  YN{E]{r) 

YN{f\i,r),  YN{f\i,r) 

REQUEST,  EPAS- 
Enhanced  REQUEST 

(Overall) 

YN^cat  (GRP,  r) , 
YN^CAT{GRP,r) 

YN(Rn){i,r),  YN^ifr) 

REQUEST,  EPAS- 
Enhanced  REQUEST 

SEX,  TSC,  EDUC 

Classification  Efficiency 

Preliminary  analyses  of  potential  classification  efficiency  benefits  of  EPAS  enhancement 
of  REQUEST  will  also  be  examined  in  the  preliminary  monthly  evaluations.  Table  5  identifies 
the  three  types  of  classification  efficiency  analysis  that  will  be  performed.  These  analyses  will  be 
based  on  predicted  performances  of  recruits  in  the  given  month.  Two  separate  predicted 
performance  values  will  be  computed  for  each  recruit  in  a  replication,  one  each  under  REQUEST 
and  EER  conditions.  The  primary  index  in  these  analyses  is  the  MPP  of  the  recruit  cohort  in  the 
evaluation  month,  computed  separately  under  the  two  assignment  conditions,  and  over 
assignment  replications  r=l,2,...,30. 

The  goal  in  these  analyses  is  the  comparison  of  overall  cohort  person-job  fit  under 
REQUEST  and  EER  classification  conditions,  as  indicated  by  their  respective  mean  predicted 
performances.  The  formal  analysis  underlying  this  comparison  may  be  carried  out  using  an 
ANOVA  framework  based  on  the  30  replicated  MPP  values  MPPR(r)  and  MPPE(r),  with  the 
REQUEST  and  EP AS-enhanced  REQUEST  conditions  as  factor  levels.  The  underlying  test 
statistic  will  account  for  the  variation  in  MPP  across  replications  under  each  simulation 
condition.  This  between  simulation  replicate  variability  will  arise  through  the  randomness  of 
MOS  training  assignments  of  the  same  recruit  cohort,  which  is  due  to  the  chance  factor  built  in 
the  choice  model.  For  the  difference  in  MPP  between  the  two  conditions  to  be  significant,  it  has 
to  be  substantial  relative  to  the  chance  factor  in  the  choice  model. 

The  analyses  will  also  be  carried  at  the  individual  MOS  level.  The  mean  predicted 
performance  of  monthly  cohort  will  be  computed  separately  for  each  MOS  in  each  replication. 
This  analysis  will  provide  information  on  how  potential  classification  gains  are  distributed  across 
MOS.  It  will  emphasize  descriptive  analysis,  identifying  MOS  with  MPP  patterns  that  depart 
substantially  from  that  in  the  overall  analysis,  rather  than  a  formal  statistical  test  of  difference  in 
the  MPP  under  the  two  conditions  by  MOS. 
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Table  5 


Classification  Effici 

iency  Analysis 

Analysis  Index 

Replicate  Observations 

Main  Analysis  Factor 
(Type  of  List) 

Subgroup 
Analysis  Factor 

Overall  MPP 

MPP„(r), 

MPPE(r) 

PP(i,  hR  ( i ,  r)) ,  PP(i,  hE  (i,  r)) 

REQUEST,  EPAS- 
Enhanced  REQUEST 

(Overall) 

MOS -Level  MPP 

MPPR(m,r), 

MPPe  (m,  r) 

PP(i,  hR  (/,  r)) ,  PP(i,  hE  (i,  r)) 

REQUEST,  EPAS- 
Enhanced  REQUEST 

(Analysis  by 

MOS) 

Classification  Efficiency  and  Size  of  Intersection 

If  the  ANOVA  analysis  of  the  overall  cohort  MPP  indicate  that  EPAS  enhancement 
provides  classification  efficiency  benefits,  a  follow-up  analysis  on  how  it  is  related  to  the  size  of 
the  intersection  between  EPAS  and  REQUEST  list  would  be  meaningful.  The  results  of  this 
analysis  may  be  useful  in  decisions  regarding  future  refinement  of  the  optimization  model 
underlying  EPAS. 

This  analysis  will  be  based  on  the  difference  in  recruit  predicted  performances  under  the 
REQUEST  and  EER  classification  conditions  given  by 

DeltaPP{i,  r )  =  PP(i ,  hE  (i,  r ))  -  PP(i,  hR  (i,  r» , 

and  on  SlZE(i,  r),  the  size  of  the  intersection  between  REQUEST  and  EPAS  lists  using  any  of 
the  three  representations  of  the  size  of  intersection.  The  formal  analysis  will  be  based  on  the 
regression  of  DeltaPP(i,  r )  on  SIZE{i,  r )  as  indicated  by  the  model 

DeltaPP(i,r )  =  50+5,x  SIZE(i,  r)+  e(i ,  r ) 

In  this  analysis  a  positively  significant  SIZE  effect  will  be  favorable  to  EPAS,  because  a  positive 
value  of  DeltaPP(i,  r )  represents  a  classification  gain  in  terms  of  predicted  performance  for  the 
ith  recruit. 

Intermediate  Analyses 

An  intermediate  set  of  analyses  will  be  carried  out  using  the  first  six  months  of  field  test 
data.  All  three  types  of  research  questions  (job  list  comparison,  classification  efficiency,  and 
accession)  will  be  examined  in  these  analyses.  Unlike  in  the  preliminary  monthly  analyses,  a 
single  result  for  each  analysis  will  be  reported  for  the  entire  six-month  evaluation  period.  The 
intermediate  analyses  will  begin  to  look  at  “cumulative  effects”  of  EPAS.  Longitudinal  patterns 
in  the  indices  that  may  develop  in  time  will  be  taken  into  account.  REQUEST  and  EER 
assignments,  however,  will  continue  to  be  based  on  the  (fixed)  actual  REQUEST  list  of  recruits. 
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What  would  be  replicated  in  the  intermediate  analysis  simulation  is  the  whole  six-month, 
rather  than  monthly,  recruit-MOS  assignments.  The  simulation  will  cycle  back  to  the  start  of  the 
first  month  at  the  end  of  the  sixth  month.  This  is  in  contrast  to  the  simulation  cycle  in  the 
preliminary  monthly  analysis,  where  recruit-MOS  assignments  are  reset  to  that  at  the  beginning 
of  the  month  after  assigning  the  last  recruit  in  the  same  month.  This  extension  from  monthly  to 
six-month  simulation  of  recruit-MOS  assignments  will  allow  cumulative  effects  to  be  examined 
in  the  intermediate  analyses. 

Extension  of  Job  List  and  Classification  Efficiency  Analyses 

Job  list  comparison  and  classification  efficiency  analyses  will  be  carried  out  in  the  same 
manner  as  described  in  the  preliminary  analysis,  but  expanded  with  the  addition  of  a  component 
representing  the  contract  month  factor.  Descriptive  analysis  comparison  of  REQUEST  and  EPAS  job 
lists,  using  the  intersection  length  YN  ( i ,  r) ,  will  look  at  possible  systematic  changes  in  monthly  mean 
lengths  of  intersection  across  replications  of  six-month  recruit  assignments.  This  will  provide  some 
insight  on  any  time-related  impact  of  EPAS  enhancement  on  recruit  MOS  training  choices. 

Analyses  of  EPAS  classification  efficiency  benefits  will  include  a  model  component  that 
will  account  for  potential  monthly  pattern  in  MPP.  This  analysis  feature,  for  example,  will  be 
useful  in  studying  the  level,  over  time,  of  anticipated  EPAS  classification  benefits.  Because  of 
the  data  generation  process  in  the  intermediate  analyses,  monthly  indices  computed  from  the 
same  replication  will  be  correlated  longitudinally.  However,  we  will  continue  to  obtain 
independent  samples  of  “longitudinal”  observations  over  replications  of  six-month  recruit 
assignments. 

Accession  Analyses 

Initial  examination  of  the  questions  related  to  monthly  accession  requirements  will  be 
added  in  the  six-month  intermediate  analyses.  We  will  compare  how  Army  accession  goals  are 
met  under  the  REQUEST  and  EER  conditions,  at  the  overall  Army  and  priority  MOS  levels. 
These  analyses,  however,  will  be  constrained  as  described  below. 

In  general,  overall  Army  accessions  will  depend  more  on  the  size  and  eligibility  of  the 
applicant  pool,  which  will  be  fixed  in  the  field  test,  than  upon  the  recruit-MOS  classification 
system.  The  analyses  of  overall  Army  accessions  will  be  restricted  in  this  sense.  However, 
differences  between  the  overall  monthly  Army  accessions  under  the  two  simulation  conditions 
are  still  possible  as  EPAS  rearranges  the  MOS-class  start  months  in  REQUEST  list.  The 
analysis  of  priority  MOS  accession  for  the  entire  evaluation  period  also  will  be  relatively  less 
constrained,  as  a  differently  rank  ordered  list  can  substantially  impact  MOS  level  fill  rates. 

Indices  required  in  accession  analysis  are  presented  in  Table  6.  In  the  overall  Army  level 
analysis,  the  proportions  of  target  accession  met  at  the  Zth  month  under  REQUEST  and  EPAS- 
enhanced  REQUEST  classification  systems,  which  are  given  by  APR(l,  r )  and  APE(l, r),  will  be 
computed  for  replications  r=l,2,...,30.  Note  that  these  indices  are  naturally  defined  relative  to 
the  receiving  station  month  rather  than  contract  month.  However,  as  in  the  job  list  and 
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classification  efficiency  analyses,  the  longitudinal  correlation  of  monthly  indices  computed  from 
the  same  replication  will  be  taken  into  account. 


Table  6 

Accession  Analysis 


Analysis  Index 

Main  Analysis  Factor 
(Type  of  List) 

Subgroup  Analysis 
Factor 

Overall  Monthly  Accession 

APR{l,r),APE(l,r) 

REQUEST,  EPAS- 
Enhanced  REQUEST 

(Overall) 

Priority  MOS  Monthly  Accession 

APr  (l,m,r),  APe  (l,m,r) 

REQUEST,  EPAS- 
Enhanced  REQUEST 

(Analysis  by  MOS) 

The  overall  Army  accession  analysis  will  be  supplemented  by  the  analyses  of  monthly 
accessions  of  priority  MOS.  This  is  important  in  uncovering  a  possible  negative  impact  of  the 
EPAS  enhancement  on  accession  goals  of  important  MOS,  which  can  occur  even  if  the  overall 
accession  analysis  indicates  equal  or  better  accession  rates  under  the  EP AS-enhanced  REQUEST 
condition.  The  proportions  of  accessions  relative  to  target  goals  of  the  mth  priority  MOS, 
represented  by  AP/(/,m,r)  and  AP/(/,m,r)  respectively  under  REQUEST  and  EP  AS-enhanced 
conditions,  will  be  computed  at  each  replication.  The  monthly  accession  pattern  across 
replications  will  be  summarized  numerically  and  graphically. 

Since  the  priority  MOS  draw  from  the  same  applicant  pool,  along  with  the  non-priority 
MOS,  accession  rates  are  not  independent  across  MOS.  (The  exact  nature  of  the  dependence 
pattern  will  be  a  characteristic  of  the  classification  system.  For  instance,  it  will  determine  the 
extent  to  which  a  gain  in  accessions  by  one  MOS  is  a  disadvantage  to  another  MOS.)  Thus,  a 
suitable  analysis  of  the  impact  of  EPAS  enhancement  on  priority  MOS  accessions  would  have  to 
compare  the  full  vector  of  priority  MOS  fill  rates  under  REQUEST  and  EP  AS-enhanced 
REQUEST  conditions.  Numerical  and  graphical  descriptive  statistics  will  be  employed  to 
highlight  fill  rate  vector  pattern  and  identify  disadvantaged  MOS  by  EPAS  enhancement. 
Univariate  test  statistics  that  will  separately  compare  accession  rates  by  MOS  under  the  two 
conditions  may  also  be  computed,  with  the  “multiple  comparison”  issue  ignored.  These 
univariate  test  statistics  can  be  utilized  less  formally  as  indices  for  identifying  disadvantaged 
MOS. 

Full  Fiscal  Year  Analyses 

The  final  set  of  analyses  will  examine  the  three  types  of  study  questions  using  a  full  fiscal 
year  evaluation  period.  These  analyses  will  employ  fixed  actual  REQUEST  lists  or  dynamically 
generated  REQUEST  lists,  depending  on  the  feasibility  of  the  latter.  The  dynamic  list  approach 
will  be  indispensable  if  alternative  DEP  lengths  are  to  be  examined.  The  simulation  will  cycle 
back  to  the  start  of  the  first  month  after  making  the  MOS  assignment  for  the  last  recruit  at  the 
12th  month  in  the  evaluation  period.  This  extended  simulation  cycle  is  expected  to  provide  better 
information  on  EPAS  cumulative  effects,  especially  if  combined  with  the  dynamic  REQUEST 
list  generation. 
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Other  than  possible  difference  in  the  type  of  REQUEST  list,  the  statistical  structure  of  the 
indices  and  their  analyses  will  be  similar  to  that  in  the  intermediate  analyses.  There  will  be 
important  issues  that  require  special  consideration  if  the  dynamic  REQUEST  list  generation  were 
employed  in  the  full  fiscal  year  simulations.  These  issues  are  discussed  below. 

Job  List  Comparison 

If  a  dynamic  list  generation  method  were  implemented,  the  composition  and  rank  order  of 
MOS-class  start  dates  would  be  shaped  dynamically  by  the  fill  rate  pattern  from  earlier  recruit 
assignments,  separately  under  REQUEST  and  EER  simulation  conditions.  In  this  case,  the 
distinction  between  assignment  and  intermediate  REQUEST  lists  mentioned  earlier  would  be 
important.  The  two  types  of  REQUEST  list  may  diverge  in  time  as  recruit-MOS  assignments  are 
made  during  the  12-month  evaluation  period.  (Their  difference  in  time  most  likely  will  depend 
on  the  length  of  DEP  allowed  in  the  simulation.)  This  is  not  a  critical  issue  in  the  preliminary  and 
intermediate  analyses,  which  are  based  on  the  fixed  actual  REQUEST  list  of  recruits,  but  needs 
to  be  considered  in  a  full  fiscal  year  analysis  that  is  based  on  the  dynamic  REQUEST  list 
approach. 

The  “TAT  and  “TP”  indices  of  the  length  of  intersection  may  be  constructed,  optionally, 
using  the  assignment  REQUEST  list.  The  analyses  based  on  these  indices  will  compare  the  MOS 
training  opportunities  that  recruits  will  face  under  the  two  classification  conditions.  Note  that  this 
is  different  from  the  analysis  of  the  “T/V”  and  “TP”  indices  that  are  constructed  using  the 
underlying  intermediate  REQUEST  list  in  the  EER  classification  condition,  which  deals  more 
with  understanding  the  extent  to  which  the  EPAS  model  optimizes  REQUEST.  Job  list  analyses 
regarding  top-n  priority  MOS  will  continue  to  use  the  assignment  version  of  REQUEST,  as  the 
relevant  underlying  issue  pertains  to  rank  ordered  MOS  training  opportunities  that  recruits  will 
face,  and  not  the  interim  rank  ordered  training  opportunities  in  the  intermediate  REQUEST  list. 

Classification  Efficiency  and  Accession  Analysis 

The  implementation  of  the  dynamic  REQUEST  list  generation  will  not  impact  the  nature 
of  analyses  for  comparing  classification  efficiency  and  accession  rates  under  REQUEST  and 
EER  conditions.  The  classification  efficiency  indices  will  be  computed  as  in  the  six-month 
intermediate  analyses.  The  longitudinal  dependence  among  monthly  values  of  these  indices, 
however,  will  remain  as  described  before. 

A  key  improvement  of  the  dynamic  list  over  the  fixed  list  approach  is  a  potentially  more 
accurate  measure  of  EPAS  cumulative  benefits.  As  mentioned  earlier,  using  dynamically  updated 
fill  rates  in  determining  the  composition  and  order  of  MOS  choices  in  the  EER  list  will  likely  to 
lead  training  assignments  that,  in  turn,  will  yield  a  fill  rate  pattern  that  is  person-job  fit  efficient 
for  subsequent  recruit  assignments. 

Based  on  the  preceding  observation,  EPAS  cumulative  efficiency  benefits  observed  in 
intermediate  analyses  using  the  fixed  REQUEST  list  may  be  viewed  as  conservative  estimates  or 
baseline  values.  Unfortunately,  this  also  implies  that  any  adverse  effect  of  EPAS  on  accession 
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goals  observed  in  analyses  based  on  a  fixed  actual  REQUEST  list  also  will  be  baseline  estimates. 
The  degree  to  which  cumulative  effects  on  accession  goals  can  be  accurately  evaluated  will  be 
determined  by  the  fidelity  of  the  simulated  model  that  will  represent  the  dynamic  factors  in  the 
REQUEST  system. 

An  additional  analysis  that  may  be  carried  out  in  the  full  fiscal  year  evaluation  is  the 
fiscal  year  accession  goals  of  priority  MOS.  The  relevant  analysis  index  and  replicate 
“observations”  are  shown  in  Table  7.  We  will  not  be  concerned  about  a  longitudinal  correlation 
in  this  analysis,  which  will  only  deal  with  the  fiscal  year  accession  rates.  However,  the  full 
vector  of  priority  MOS  accession  rates  will  continue  to  be  correlated  as  the  MOS  compete  from 
the  same  pool  of  applicants.  Thus,  comparison  between  accession  rates  under  the  REQUEST  and 
EER  conditions  will  be  carried  out  in  a  multivariate  fashion. 

Table  7 


Expanded  Accession  Analysis 


Analysis  Index 

. . . . 

Replicate  Observations 

Main  Analysis  Factor 

Subgroup 

Overall  Monthly  Accession 

AP,(l),  APc(l) 

APR(l,r),APE{l,r) 

REQUEST,  EPAS- 
Enhanced  REQUEST  Lists 

(Overall) 

Priority  MOS  Monthly  Accession 

APR{l,m),  APE (l,m) 

APr  (/,  m,  r ) ,  APg  (/,  m,  r) 

REQUEST,  EPAS- 
Enhanced  REQUEST  Lists 

(MOS  Level) 

Priority  MOS  Fiscal  Year  Accession 

AP,n  M,  ap;» 

APRr (m,r),  APEY {m, r) 

REQUEST,  EPAS- 
Enhanced  REQUEST  Lists 

(MOS  Level) 
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Appendix  A 

Overview  of  Recruit  MOS  Assignment  Decision  Model  Approach 

In  this  write-up  we  give  a  technical  overview  of  our  approach  to  modeling  recruit  job 
choice  behavior.  The  technique  that  we  present  here  is  part  of  what  is  commonly  known  in  the 
econometrics  literature  as  discrete  choice  modeling.11  This  will  be  described  using  an  Army 
recruiting  framework.  Our  discussion  will  focus  on  the  rationale  of  the  general  approach,  using  a 
simplified  version  of  the  model.  Complete  specification  of  the  choice  model  will  be  finalized 
using  actual  REQUEST  transaction  data.  We  also  outline  how  the  choice  model  will  be  utilized 
for  simulating  training  choices  of  recruits  in  the  off-line  MOS  assignments.  The  model  obtained 
at  the  end  is  not  intended  to  represent  an  individual  recruit’s  choice  decision  process.  The 
objective  is  to  statistically  model  MOS  choice  and  recruit  profile  pattern  in  the  population  for  the 
purpose  of  simulating  recruit  choice  in  the  EPAS  field  test. 

Consider  a  hypothetical  transaction  at  a  Military  Enlistment  Processing  Site  (MEPS) 
during  which  a  new  Army  recruit  is  presented  with  a  list  of  training  choices.  The  index  i  will  be 
used  to  label  recruits  and  index  j  to  label  the  choices  in  the  list  presented  to  the  ith  recruit.  We 
simplify  our  discussion  by  assuming  that  a  fixed  set  of  m  distinct  MOS  training  choices  are 
presented  to  all  recruits.  In  actual  MEPS  transactions,  the  same  MOS  may  appear  twice  or  more 
with  different  training  start  dates. 

We  now  characterize  the  enlistment  decision  problem  of  the  ith  recruit  who  is  presented 
with  m  alternative  MOS  training  choices.  From  the  perspective  of  this  particular  recruit,  the  choice 
may  be  determined  by  the  utilities  that  he  or  she  attaches  to  the  alternative  MOS.  For  recruit  choice 
modeling  purposes,  utility  is  associated  to  a  general  notion  of  “attraction”  that  may  or  may  not  be 
economic  in  nature.  Using  this  relaxed  notion  of  utility,  it  may  be  assumed  that  the  recruit  will 
behave  rationally  by  picking  the  training  option  in  the  list  that  he  or  she  “likes”  best.  More 
formally,  let  {UiVUi2,...,Uim }  denote  the  utility  set  that  the  ith  recruit  associates  to  these  m  choices. 

If  Uc.  =  max \Uy\j  =  l,2,...,m},  then  the  recruit  will  choose  the  c,. th  training  option  in  the  list. 

The  individual-centric  choice  behavior  described  above  is  fully  deterministic.  Given  a  set 
of  MOS  training  choices  (with  specified  choice  attributes  such  as  start  dates,  bonuses  and 
incentives),  we  expect  the  same  recruit  to  choose  the  same  training  option.  However,  we  are  not 
interested  in  the  behavior  of  an  individual  recruit  per  se,  but  in  the  recruit  profile-MOS  choice 
pattern  revealed  in  the  MEPS  transaction  data.  In  other  words,  we  want  to  be  able  to  model 
recruit  choice  behavior  from  the  perspective  of  the  researcher.  To  accomplish  this,  we  let 


1 1  McFadden,  D.  (1974).  Conditional  Logit  Analysis  of  Qualitative  Choice  Behavior.  In  P.  Zarembka  (Ed.). 
Frontiers  in  Econometrics.  New  York:  Academic  Press. 
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represent  the  utility  of  the  ith  recruit  for  the  yth  training  option.  The  first  term  on  the  right  hand 
side  is  of  the  form  Vi}  =  /  (x,  ,Zy)-  The  value  of  /  (x,. ,  Zy  )  represents  the  common  level  of 

attractiveness  of  all  recruits  in  the  population  with  attribute  vector  x;  for  the  yth  training  option 
with  attribute  vector  Zy .  The  second  term,  Ei} ,  represents  idiosyncrasies  of  individual  “tastes” 
that  are  unobservable  from  the  researcher’s  point  of  view.  It  is  from  the  second  component  that 
the  “random”  part  in  our  choice  model  is  obtained.  The  EtJ  s  are  treated  as  realizations  from 

some  probability  distribution. 

In  our  approach  we  will  take/to  be  a  linear  function  of  recruit  and  choice  attributes.  We 
will  write 


Vy=aTXi+bTZy 

where  a  and  b  are  coefficient  vectors  associated,  respectively,  to  recruit  attributes  and  training 
choice  attributes.  The  coefficients  will  reflect  the  importance  of  the  associated  recruit  or  choice 
attribute  in  the  determination  of  utility  Uy .  Under  simple  distribution  assumptions  for  Etj ,  the 

probability  piq  that  the  ith  recruit  will  select  the  q\h  training  choice,  from  a  researcher’s  point  of 
view,  is  given  by 


exp(y,? ) 

- —  > 

2>p(n) 

H 

for  q=l,2,...,m.  This  is  also  known  as  the  multinomial  logit  discrete  choice  model. 

The  parameter  vectors  of  coefficients,  a  and  b,  of  specified  recruit  and  choice  attribute 
sets  may  be  estimated  from  REQUEST  transaction  data  given  the  above  probability  model 
representation. 

The  parameter  estimates  then  will  be  used  to  compute  probabilities  [py\j  =  l,2,...,m|  for 

the  ith  individual  in  the  field  test.  The  probability  Py  will  represent  the  likelihood  of  the  ith 

recruit  “choosing”  the  jth  training  option  in  his  or  her  REQUEST  list  during  the  off-line 
simulations.  In  this  probabilistic  representation  of  choice  behavior,  the  ith  recruit  does  not 
necessarily  select  the  choice  corresponding  to  the  largest  probability.  (The  probabilities  are  a 
researcher’s  representation  of  the  recruit’s  choice  behavior  pattern,  and  are  not  to  be  equated  to 
the  individual  recruit’s  utilities.)  Instead,  the  probabilities  will  represent  the  odds  of  the 
alternative  training  opportunities  in  the  list,  any  of  which  can  be  randomly  selected  by  the  recruit 
at  the  rth  simulation  replication.  These  probabilities  will  reflect  the  relative  proportions  of 
selection  over  infinite  replications  of  recruit  assignments. 

We  briefly  illustrate  a  randomized  recruit  choice  selection  process  that  will  be  considered 
in  the  simulations.  To  simplify  the  example,  we  suppose  that  a  recruit  is  presented  with  only  m= 5 
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training  options.  Based  on  the  estimates  of  the  choice  model  parameters,  suppose  we  computed 
the  probabilities  that  the  recruit  will  select  each  of  the  five  training  choices  to  be  .  10,  .10,  .30, 
.35,  and  .15,  respectively.  A  randomization  rule  that  can  be  employed  during  the  rth  replication 
is  as  follows.  Generate  a  uniform  pseudo  random  number  W between  0  and  1.  Then  assign  the 
recruit  to  the  ;'th  alternative  training  opportunity  in  the  REQUEST  list  according  to  the  table 


Value  of  W 

0<W<..10 

.10<W<.20 

,20<W<.50 

.50<w<;.85 

.85<W<.1.00 

Training  Choice 

7=1 

_ M _ 

_ M _ 

J= 4 

_ M _ 

This  method  will  yield  random  assignments  for  the  recruit  across  replications  with 
expected  proportions  given  by  .10,  .10,  .30,  .35,  and  .15. 


A-3 


Appendix  B 

Quality  of  the  Intersection  Between  the  REQUEST  and  EPAS  Lists 

The  indices  based  on  simple  counts  do  not  take  into  account  the  rank  order  location  of 
REQUEST  MOS-class  start  dates  in  the  EPAS  list.  Instead,  all  matching  MOS-class  start  dates 
contribute  equally  to  the  values  of  the  indices.  Consider,  for  example,  two  recruits  (/=  1 ,2)  with 
actual  counts  YN( 1,  r )  and  YN(2,r)  of  matching  MOS-class  dates  in  their  respective  REQUEST 
and  EPAS  lists.  Now  KV(l,  r)  could  be  equal  to  YN(2,  r)  even  if  all  of  the  matching  MOS-class 
dates  for  the  first  recruit  appear  at  the  top-half  of  his  EPAS  list,  while  those  for  the  second  recruit 
appear  at  the  bottom-half  of  his  EPAS  list.  Thus,  the  similarities  between  REQUEST  and  EPAS 
lists  of  these  two  recruits  implicitly  are  equal  using  the  index  YN.  In  turn,  this  would  seem  to 
suggest  that  the  two  recruits  contribute  equally  to  EPAS  enhancement  of  REQUEST.  This 
unintended  interpretation  clearly  is  not  desirable. 

We  address  the  preceding  limitation  by  specifying  an  index  that  takes  into  account  (1)  the 
rank  order  location  of  matching  MOS-class  dates  in  the  EPAS  list  and  (2)  the  changes  in  their 
rank  order  positions  from  the  original  REQUEST  list  to  the  EP AS-enhanced  REQUEST  list. 

This  measure  of  similarity  will  be  based  on  a  “quality  score”  Q  which  we  construct  for  each 
recruit  as  follows. 

Start  by  using  EER(m,i,r )  to  denote  the  EP  AS-enhanced  REQUEST  rank  order  position 
of  the  mth  MOS-class  date  alternative  in  the  REQUEST  list  of  the  ith  recruit.  Then 

DR(m,i,r)  =  m-EER(m,i,r ) 


is  the  change  in  the  rank  order  position  of  the  mth  MOS-class  date  in  the  REQUEST  list  after 
EPAS  reordering  of  the  REQUEST  list.  We  also  record  the  EPAS  list  rank  order  location  of  each 
alternative  MOS-class  date  in  REQUEST  by  letting 


ER(m,i,r)  = 


''rank  order  of  the  EPAS  MOS  -  class  month  that  matches^ 
^the  mth  REQUEST  MOS-class  date  j 


if  the  mth  MOS-class  date  in  the  REQUEST  list  appears  in  the  EPAS  list;  otherwise,  we  let 
ER(m,i,r )  =  -1  to  indicate  that  the  mth  MOS-class  date  is  not  in  the  EPAS  list. 

Using  the  preceding  expressions,  the  contribution  of  the  mth  MOS-class  date  alternative 
in  REQUEST  list  may  be  quantified  meaningfully  by 
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QM(m,i,r ) 


DR(m,i,r ) 
LR(i,r ) 

-1 

0 


1  + 


1- 


ER(m,i,r) 

LE(i,r) 


ER(m,  i,r)  >  0  and  DR(m, i,r)  >  0 

ER(m,  i,  r)  >  0  and  DR(m,  i,  r)  <  0 
ER(m,i,r)=  - 1 


where  LE(i,r)  is  equal  to  the  length  of  the  EPAS  list  of  the  ith  recruit.  An  intersection  quality 
value  computed  for  the  ith  recruit  based  on  r)  is  given  by 

QP(i,r)=[\/LR(i,r)]  £ QM{m,i,r ) 

m= 1 

We  make  the  following  initial  observations  on  the  proposed  quality  value  QP(i,r).  First, 
the  difference  [l  -  ER (m,i,r)/ LE (i,  r )]  represents  the  EPAS  percentile  rank  of  the  mth  MOS. 
(Percentile  rank  values  near  1  correspond  to  MOS-class  start  dates  at  the  top  of  the  EPAS  list.) 
The  fraction  DR(m,i,r)/ LR(i,r)  represents  the  REQUEST  percentile  rank  difference  associated 
with  the  mth  MOS-class  start  date  due  to  EPAS  reordering  of  the  REQUEST  list.  Thus,  using  the 
first  line  on  the  right-hand  side  of  the  expression  for  QM (m,i,r)  above,  REQUEST  MOS-class 
dates  that  match  MOS-class  months  at  the  top  of  the  EPAS  list  will  have  high  contributions  to 
QP(i,r).  Moreover,  these  MOS-class  dates  will  have  higher  contributions  to  the  intersection 
quality  value  if  their  rank  orders  go  up  substantially  after  EPAS  reordering. 

Continuing,  we  look  at  the  second  line  in  the  expression  for  QM(m,i,r).  This  will 
correspond  to  the  quality  value  contribution  of  MOS-class  dates  in  the  REQUEST  list  with 
matching  EPAS-class  months,  but  whose  rank  order  dropped  after  EPAS  enhancement. 
QM(m,i,r )  does  not  penalize  these  MOS-class  dates.  They  will  continue  to  contribute  to  the 
intersection  quality  score,  but  only  by  unity  (as  in  the  simple  count  indices).  The  drop  in  rank 
order  can  only  occur  if  there  is  at  least  one  MOS-class  date  that  should  be  ranked  higher. 

It  can  be  verified  that  YP(i,r )  <  QP{i,r)  <  2 YP(i,r) ,  where  YP(i,r)  is  the  proportion  of 
common  MOS-class  dates  relative  to  the  length  of  the  REQUEST  list,  as  defined  earlier.  The 
lower  limit  equality  occurs  if  all  MOS-class  dates  in  the  REQUEST  list  with  matching  EPAS 
class-months  appear  at  the  top-IW(/,r)  of  the  list  in  proper  EPAS-order.  Under  this  condition 
EPAS  does  not  enhance  the  REQUEST  list  as  it  is  already  in  the  desired  EPAS  order,  and  the 
quality  of  intersection  is  nothing  but  the  size  of  intersection  measured  in  proportion  (i.e., 
YP(i,r)).  On  the  other  hand,  QP(i,r )  is  near  the  upper  bound  2 YP(i,r)  when  the  REQUEST 
MOS-class  dates  match  the  top-I7V(/,r)  MOS-class  months  in  the  EPAS  list,  but  appear  at  the 
bottom-  YN(i,r)  of  the  REQUEST  list  in  reversed  EPAS-  order. 

The  intersection  quality  value  QP(i,r)  is  an  improvement  over  YP(i,r),  the  proportion 
index  based  on  simple  count  of  matching  MOS  training  opportunities  in  the  REQUEST  and 
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EPAS  lists  of  recruits.  A  meaningful  measure  of  the  contribution  of  the  ith  recruit  to  the  overall 
EPAS  enhancement  of  REQUEST  is  given  by  QP(i,r)-YP(i,r) .  This  difference  expression 
separates  the  enhancement  of  REQUEST  due  to  EPAS  reordering  from  the  simple  count  of 
matching  MOS  training  class  dates.  The  overall  or  subgroup  forms  of  the  difference  QP  -  YP 
would  be  useful  in  characterizing  the  potential  EPAS  enhancement. 


